+
+ + Computation and Language 104 +
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ + ☆ Context Canvas: Enhancing Text-to-Image Diffusion Models with Knowledge + Graph-Based RAG +
+ +
+ We introduce a novel approach to enhance the capabilities of text-to-image
+models by incorporating a graph-based RAG. Our system dynamically retrieves
+detailed character information and relational data from the knowledge graph,
+enabling the generation of visually accurate and contextually rich images. This
+capability significantly improves upon the limitations of existing T2I models,
+which often struggle with the accurate depiction of complex or culturally
+specific subjects due to dataset constraints. Furthermore, we propose a novel
+self-correcting mechanism for text-to-image models to ensure consistency and
+fidelity in visual outputs, leveraging the rich context from the graph to guide
+corrections. Our qualitative and quantitative experiments demonstrate that
+Context Canvas significantly enhances the capabilities of popular models such
+as Flux, Stable Diffusion, and DALL-E, and improves the functionality of
+ControlNet for fine-grained image editing tasks. To our knowledge, Context
+Canvas represents the first application of graph-based RAG in enhancing T2I
+models, representing a significant advancement for producing high-fidelity,
+context-aware multi-faceted images.
+
+
+
+ comment: Project Page: https://context-canvas.github.io/
+
+
+
+ + ☆ Olympus: A Universal Task Router for Computer Vision Tasks +
+ +
+ We introduce Olympus, a new approach that transforms Multimodal Large
+Language Models (MLLMs) into a unified framework capable of handling a wide
+array of computer vision tasks. Utilizing a controller MLLM, Olympus delegates
+over 20 specialized tasks across images, videos, and 3D objects to dedicated
+modules. This instruction-based routing enables complex workflows through
+chained actions without the need for training heavy generative models. Olympus
+easily integrates with existing MLLMs, expanding their capabilities with
+comparable performance. Experimental results demonstrate that Olympus achieves
+an average routing accuracy of 94.75% across 20 tasks and precision of 91.82%
+in chained action scenarios, showcasing its effectiveness as a universal task
+router that can solve a diverse range of computer vision tasks. Project page:
+https://github.com/yuanze-lin/Olympus_page
+
+
+
+ comment: Technical Report
+
+
+
+ + ☆ AgentTrek: Agent Trajectory Synthesis via Guiding Replay with Web + Tutorials +
+
+
+
+
+
+
+
+ Yiheng Xu, Dunjie Lu, Zhennan Shen, Junli Wang, Zekun Wang, Yuchen Mao, Caiming Xiong, Tao Yu
+
+
+ Graphical User Interface (GUI) agents hold great potential for automating
+complex tasks across diverse digital environments, from web applications to
+desktop software. However, the development of such agents is hindered by the
+lack of high-quality, multi-step trajectory data required for effective
+training. Existing approaches rely on expensive and labor-intensive human
+annotation, making them unsustainable at scale. To address this challenge, we
+propose AgentTrek, a scalable data synthesis pipeline that generates
+high-quality GUI agent trajectories by leveraging web tutorials. Our method
+automatically gathers tutorial-like texts from the internet, transforms them
+into task goals with step-by-step instructions, and employs a visual-language
+model agent to simulate their execution in a real digital environment. A
+VLM-based evaluator ensures the correctness of the generated trajectories. We
+demonstrate that training GUI agents with these synthesized trajectories
+significantly improves their grounding and planning performance over the
+current models. Moreover, our approach is more cost-efficient compared to
+traditional human annotation methods. This work underscores the potential of
+guided replay with web tutorials as a viable strategy for large-scale GUI agent
+training, paving the way for more capable and autonomous digital agents.
+
+
+
+ comment: https://agenttrek.github.io
+
+
+
+ + ☆ TimeRefine: Temporal Grounding with Time Refining Video LLM +
+
+
+
+
+
+
+
+ Xizi Wang, Feng Cheng, Ziyang Wang, Huiyu Wang, Md Mohaiminul Islam, Lorenzo Torresani, Mohit Bansal, Gedas Bertasius, David Crandall
+
+
+ Video temporal grounding aims to localize relevant temporal boundaries in a
+video given a textual prompt. Recent work has focused on enabling Video LLMs to
+perform video temporal grounding via next-token prediction of temporal
+timestamps. However, accurately localizing timestamps in videos remains
+challenging for Video LLMs when relying solely on temporal token prediction.
+Our proposed TimeRefine addresses this challenge in two ways. First, instead of
+directly predicting the start and end timestamps, we reformulate the temporal
+grounding task as a temporal refining task: the model first makes rough
+predictions and then refines them by predicting offsets to the target segment.
+This refining process is repeated multiple times, through which the model
+progressively self-improves its temporal localization accuracy. Second, to
+enhance the model's temporal perception capabilities, we incorporate an
+auxiliary prediction head that penalizes the model more if a predicted segment
+deviates further from the ground truth, thus encouraging the model to make
+closer and more accurate predictions. Our plug-and-play method can be
+integrated into most LLM-based temporal grounding approaches. The experimental
+results demonstrate that TimeRefine achieves 3.6% and 5.0% mIoU improvements on
+the ActivityNet and Charades-STA datasets, respectively. Code and pretrained
+models will be released.
+
+
+
+
+ + ☆ InternLM-XComposer2.5-OmniLive: A Comprehensive Multimodal System for + Long-term Streaming Video and Audio Interactions +
+
+
+
+
+
+
+
+ Pan Zhang, Xiaoyi Dong, Yuhang Cao, Yuhang Zang, Rui Qian, Xilin Wei, Lin Chen, Yifei Li, Junbo Niu, Shuangrui Ding, Qipeng Guo, Haodong Duan, Xin Chen, Han Lv, Zheng Nie, Min Zhang, Bin Wang, Wenwei Zhang, Xinyue Zhang, Jiaye Ge, Wei Li, Jingwen Li, Zhongying Tu, Conghui He, Xingcheng Zhang, Kai Chen, Yu Qiao, Dahua Lin, Jiaqi Wang
+
+
+ Creating AI systems that can interact with environments over long periods,
+similar to human cognition, has been a longstanding research goal. Recent
+advancements in multimodal large language models (MLLMs) have made significant
+strides in open-world understanding. However, the challenge of continuous and
+simultaneous streaming perception, memory, and reasoning remains largely
+unexplored. Current MLLMs are constrained by their sequence-to-sequence
+architecture, which limits their ability to process inputs and generate
+responses simultaneously, akin to being unable to think while perceiving.
+Furthermore, relying on long contexts to store historical data is impractical
+for long-term interactions, as retaining all information becomes costly and
+inefficient. Therefore, rather than relying on a single foundation model to
+perform all functions, this project draws inspiration from the concept of the
+Specialized Generalist AI and introduces disentangled streaming perception,
+reasoning, and memory mechanisms, enabling real-time interaction with streaming
+video and audio input. The proposed framework InternLM-XComposer2.5-OmniLive
+(IXC2.5-OL) consists of three key modules: (1) Streaming Perception Module:
+Processes multimodal information in real-time, storing key details in memory
+and triggering reasoning in response to user queries. (2) Multi-modal Long
+Memory Module: Integrates short-term and long-term memory, compressing
+short-term memories into long-term ones for efficient retrieval and improved
+accuracy. (3) Reasoning Module: Responds to queries and executes reasoning
+tasks, coordinating with the perception and memory modules. This project
+simulates human-like cognition, enabling multimodal large language models to
+provide continuous and adaptive service over time.
+
+
+
+ comment: Github Repo:
+ https://github.com/InternLM/InternLM-XComposer/tree/main/InternLM-XComposer-2.5-OmniLive
+
+
+
+ + ☆ OpenNER 1.0: Standardized Open-Access Named Entity Recognition Datasets + in 50+ Languages +
+
+
+
+
+
+
+
+ Chester Palen-Michel, Maxwell Pickering, Maya Kruse, Jonne Sälevä, Constantine Lignos
+
+
+ We present OpenNER 1.0, a standardized collection of openly available named
+entity recognition (NER) datasets. OpenNER contains 34 datasets spanning 51
+languages, annotated in varying named entity ontologies. We correct annotation
+format issues, standardize the original datasets into a uniform representation,
+map entity type names to be more consistent across corpora, and provide the
+collection in a structure that enables research in multilingual and
+multi-ontology NER. We provide baseline models using three pretrained
+multilingual language models to compare the performance of recent models and
+facilitate future research in NER.
+
+
+
+
+ + ☆ DISHONEST: Dissecting misInformation Spread using Homogeneous sOcial + NEtworks and Semantic Topic classification +
+ +
+ The emergence of the COVID-19 pandemic resulted in a significant rise in the
+spread of misinformation on online platforms such as Twitter. Oftentimes this
+growth is blamed on the idea of the "echo chamber." However, the behavior said
+to characterize these echo chambers exists in two dimensions. The first is in a
+user's social interactions, where they are said to stick with the same clique
+of like-minded users. The second is in the content of their posts, where they
+are said to repeatedly espouse homogeneous ideas. In this study, we link the
+two by using Twitter's network of retweets to study social interactions and
+topic modeling to study tweet content. In order to measure the diversity of a
+user's interactions over time, we develop a novel metric to track the speed at
+which they travel through the social network. The application of these analysis
+methods to misinformation-focused data from the pandemic demonstrates
+correlation between social behavior and tweet content. We believe this
+correlation supports the common intuition about how antisocial users behave,
+and further suggests that it holds even in subcommunities already rife with
+misinformation.
+
+
+
+
+ + ☆ DiverseAgentEntropy: Quantifying Black-Box LLM Uncertainty through + Diverse Perspectives and Multi-Agent Interaction +
+
+
+
+
+
+
+
+ Yu Feng, Phu Mon Htut, Zheng Qi, Wei Xiao, Manuel Mager, Nikolaos Pappas, Kishaloy Halder, Yang Li, Yassine Benajiba, Dan Roth
+
+
+ Quantifying the uncertainty in the factual parametric knowledge of Large
+Language Models (LLMs), especially in a black-box setting, poses a significant
+challenge. Existing methods, which gauge a model's uncertainty through
+evaluating self-consistency in responses to the original query, do not always
+capture true uncertainty. Models might respond consistently to the origin query
+with a wrong answer, yet respond correctly to varied questions from different
+perspectives about the same query, and vice versa. In this paper, we propose a
+novel method, DiverseAgentEntropy, for evaluating a model's uncertainty using
+multi-agent interaction under the assumption that if a model is certain, it
+should consistently recall the answer to the original query across a diverse
+collection of questions about the same original query. We further implement an
+abstention policy to withhold responses when uncertainty is high. Our method
+offers a more accurate prediction of the model's reliability and further
+detects hallucinations, outperforming other self-consistency-based methods.
+Additionally, it demonstrates that existing models often fail to consistently
+retrieve the correct answer to the same query under diverse varied questions
+even when knowing the correct answer.
+
+
+
+
+ + ☆ JuStRank: Benchmarking LLM Judges for System Ranking +
+ +
+ Given the rapid progress of generative AI, there is a pressing need to
+systematically compare and choose between the numerous models and
+configurations available. The scale and versatility of such evaluations make
+the use of LLM-based judges a compelling solution for this challenge.
+Crucially, this approach requires first to validate the quality of the LLM
+judge itself. Previous work has focused on instance-based assessment of LLM
+judges, where a judge is evaluated over a set of responses, or response pairs,
+while being agnostic to their source systems. We argue that this setting
+overlooks critical factors affecting system-level ranking, such as a judge's
+positive or negative bias towards certain systems. To address this gap, we
+conduct the first large-scale study of LLM judges as system rankers. System
+scores are generated by aggregating judgment scores over multiple system
+outputs, and the judge's quality is assessed by comparing the resulting system
+ranking to a human-based ranking. Beyond overall judge assessment, our analysis
+provides a fine-grained characterization of judge behavior, including their
+decisiveness and bias.
+
+
+
+
+ + ★ Does Representation Matter? Exploring Intermediate Layers in Large + Language Models +
+ +
+ Understanding what defines a good representation in large language models
+(LLMs) is fundamental to both theoretical understanding and practical
+applications. In this paper, we investigate the quality of intermediate
+representations in various LLM architectures, including Transformers and State
+Space Models (SSMs). We find that intermediate layers often yield more
+informative representations for downstream tasks than the final layers. To
+measure the representation quality, we adapt and apply a suite of metrics -
+such as prompt entropy, curvature, and augmentation-invariance - originally
+proposed in other contexts. Our empirical study reveals significant
+architectural differences, how representations evolve throughout training, and
+how factors like input randomness and prompt length affect each layer. Notably,
+we observe a bimodal pattern in the entropy of some intermediate layers and
+consider potential explanations tied to training data. Overall, our results
+illuminate the internal mechanics of LLMs and guide strategies for
+architectural optimization and training.
+
+
+
+ comment: Accepted to 2024 NeurIPs Workshop on Machine Learning and Compression
+
+
+
+ + ☆ Foundational Large Language Models for Materials Research +
+
+
+
+
+
+
+
+ Vaibhav Mishra, Somaditya Singh, Dhruv Ahlawat, Mohd Zaki, Vaibhav Bihani, Hargun Singh Grover, Biswajit Mishra, Santiago Miret, Mausam, N. M. Anoop Krishnan
+
+
+ Materials discovery and development are critical for addressing global
+challenges. Yet, the exponential growth in materials science literature
+comprising vast amounts of textual data has created significant bottlenecks in
+knowledge extraction, synthesis, and scientific reasoning. Large Language
+Models (LLMs) offer unprecedented opportunities to accelerate materials
+research through automated analysis and prediction. Still, their effective
+deployment requires domain-specific adaptation for understanding and solving
+domain-relevant tasks. Here, we present LLaMat, a family of foundational models
+for materials science developed through continued pretraining of LLaMA models
+on an extensive corpus of materials literature and crystallographic data.
+Through systematic evaluation, we demonstrate that LLaMat excels in
+materials-specific NLP and structured information extraction while maintaining
+general linguistic capabilities. The specialized LLaMat-CIF variant
+demonstrates unprecedented capabilities in crystal structure generation,
+predicting stable crystals with high coverage across the periodic table.
+Intriguingly, despite LLaMA-3's superior performance in comparison to LLaMA-2,
+we observe that LLaMat-2 demonstrates unexpectedly enhanced domain-specific
+performance across diverse materials science tasks, including structured
+information extraction from text and tables, more particularly in crystal
+structure generation, a potential adaptation rigidity in overtrained LLMs.
+Altogether, the present work demonstrates the effectiveness of domain
+adaptation towards developing practically deployable LLM copilots for materials
+research. Beyond materials science, our findings reveal important
+considerations for domain adaptation of LLMs, such as model selection, training
+methodology, and domain-specific performance, which may influence the
+development of specialized scientific AI systems.
+
+
+
+
+ + ☆ Audios Don't Lie: Multi-Frequency Channel Attention Mechanism for Audio + Deepfake Detection +
+ +
+ With the rapid development of artificial intelligence technology, the
+application of deepfake technology in the audio field has gradually increased,
+resulting in a wide range of security risks. Especially in the financial and
+social security fields, the misuse of deepfake audios has raised serious
+concerns. To address this challenge, this study proposes an audio deepfake
+detection method based on multi-frequency channel attention mechanism (MFCA)
+and 2D discrete cosine transform (DCT). By processing the audio signal into a
+melspectrogram, using MobileNet V2 to extract deep features, and combining it
+with the MFCA module to weight different frequency channels in the audio
+signal, this method can effectively capture the fine-grained frequency domain
+features in the audio signal and enhance the Classification capability of fake
+audios. Experimental results show that compared with traditional methods, the
+model proposed in this study shows significant advantages in accuracy,
+precision,recall, F1 score and other indicators. Especially in complex audio
+scenarios, this method shows stronger robustness and generalization
+capabilities and provides a new idea for audio deepfake detection and has
+important practical application value. In the future, more advanced audio
+detection technologies and optimization strategies will be explored to further
+improve the accuracy and generalization capabilities of audio deepfake
+detection.
+
+
+
+
+ + ☆ The Impact of Copyrighted Material on Large Language Models: A Norwegian + Perspective +
+
+
+
+
+
+
+
+ Javier de la Rosa, Vladislav Mikhailov, Lemei Zhang, Freddy Wetjen, David Samuel, Peng Liu, Rolv-Arild Braaten, Petter Mæhlum, Magnus Breder Birkenes, Andrey Kutuzov, Tita Enstad, Svein Arne Brygfjeld, Jon Atle Gulla, Stephan Oepen, Erik Velldal, Wilfred Østgulen, Liljia Øvrelid, Aslak Sira Myhre
+
+
+ The use of copyrighted materials in training generative language models
+raises critical legal and ethical questions. This paper presents a framework
+for and the results of empirically assessing the impact of copyrighted
+materials on the performance of large language models (LLMs) for Norwegian. We
+found that both books and newspapers contribute positively when the models are
+evaluated on a diverse set of Norwegian benchmarks, while fiction works
+possibly lead to decreased performance. Our experiments could inform the
+creation of a compensation scheme for authors whose works contribute to AI
+development.
+
+
+
+ comment: pre-print, under review
+
+
+
+ + ☆ From Intention To Implementation: Automating Biomedical Research via + LLMs +
+ +
+ Conventional biomedical research is increasingly labor-intensive due to the
+exponential growth of scientific literature and datasets. Artificial
+intelligence (AI), particularly Large Language Models (LLMs), has the potential
+to revolutionize this process by automating various steps. Still, significant
+challenges remain, including the need for multidisciplinary expertise,
+logicality of experimental design, and performance measurements. This paper
+introduces BioResearcher, the first end-to-end automated system designed to
+streamline the entire biomedical research process involving dry lab
+experiments. BioResearcher employs a modular multi-agent architecture,
+integrating specialized agents for search, literature processing, experimental
+design, and programming. By decomposing complex tasks into logically related
+sub-tasks and utilizing a hierarchical learning approach, BioResearcher
+effectively addresses the challenges of multidisciplinary requirements and
+logical complexity. Furthermore, BioResearcher incorporates an LLM-based
+reviewer for in-process quality control and introduces novel evaluation metrics
+to assess the quality and automation of experimental protocols. BioResearcher
+successfully achieves an average execution success rate of 63.07% across eight
+previously unmet research objectives. The generated protocols averagely
+outperform typical agent systems by 22.0% on five quality metrics. The system
+demonstrates significant potential to reduce researchers' workloads and
+accelerate biomedical discoveries, paving the way for future innovations in
+automated research systems.
+
+
+
+
+ + ☆ Unifying AI Tutor Evaluation: An Evaluation Taxonomy for Pedagogical + Ability Assessment of LLM-Powered AI Tutors +
+ +
+ In this paper, we investigate whether current state-of-the-art large language
+models (LLMs) are effective as AI tutors and whether they demonstrate
+pedagogical abilities necessary for good AI tutoring in educational dialogues.
+Previous efforts towards evaluation have been limited to subjective protocols
+and benchmarks. To bridge this gap, we propose a unified evaluation taxonomy
+with eight pedagogical dimensions based on key learning sciences principles,
+which is designed to assess the pedagogical value of LLM-powered AI tutor
+responses grounded in student mistakes or confusion in the mathematical domain.
+We release MRBench -- a new evaluation benchmark containing 192 conversations
+and 1,596 responses from seven state-of-the-art LLM-based and human tutors,
+providing gold annotations for eight pedagogical dimensions. We assess
+reliability of the popular Prometheus2 LLM as an evaluator and analyze each
+tutor's pedagogical abilities, highlighting which LLMs are good tutors and
+which ones are more suitable as question-answering systems. We believe that the
+presented taxonomy, benchmark, and human-annotated labels will streamline the
+evaluation process and help track the progress in AI tutors' development.
+
+
+
+ comment: 8 pages
+
+
+
+ + ☆ Text Generation Models for Luxembourgish with Limited Data: A Balanced + Multilingual Strategy +
+ +
+ This paper addresses the challenges in developing language models for
+less-represented languages, with a focus on Luxembourgish. Despite its active
+development, Luxembourgish faces a digital data scarcity, exacerbated by
+Luxembourg's multilingual context. We propose a novel text generation model
+based on the T5 architecture, combining limited Luxembourgish data with equal
+amounts, in terms of size and type, of German and French data. We hypothesise
+that a model trained on Luxembourgish, German, and French will improve the
+model's cross-lingual transfer learning capabilities and outperform monolingual
+and large multilingual models. To verify this, the study at hand explores
+whether multilingual or monolingual training is more beneficial for
+Luxembourgish language generation. For the evaluation, we introduce LuxGen, a
+text generation benchmark that is the first of its kind for Luxembourgish.
+
+
+
+ comment: Accepted at VarDial 2025
+
+
+
+ + ☆ Imitate, Explore, and Self-Improve: A Reproduction Report on + Slow-thinking Reasoning Systems +
+
+
+
+
+
+
+
+ Yingqian Min, Zhipeng Chen, Jinhao Jiang, Jie Chen, Jia Deng, Yiwen Hu, Yiru Tang, Jiapeng Wang, Xiaoxue Cheng, Huatong Song, Wayne Xin Zhao, Zheng Liu, Zhongyuan Wang, Ji-Rong Wen
+
+
+ Recently, slow-thinking reasoning systems, such as o1, have demonstrated
+remarkable capabilities in solving complex reasoning tasks. These systems
+typically engage in an extended thinking process before responding to a query,
+allowing them to generate more thorough, accurate, and well-reasoned solutions.
+These systems are primarily developed and maintained by industry, with their
+core techniques not publicly disclosed. In response, an increasing number of
+studies from the research community aim to explore the technical foundations
+underlying these powerful reasoning systems. Building on these prior efforts,
+this paper presents a reproduction report on implementing o1-like reasoning
+systems. We introduce an "imitate, explore, and self-improve" framework as our
+primary technical approach to train the reasoning model. In the initial phase,
+we use distilled long-form thought data to fine-tune the reasoning model,
+enabling it to invoke a slow-thinking mode. The model is then encouraged to
+explore challenging problems by generating multiple rollouts, which can result
+in increasingly more high-quality trajectories that lead to correct answers.
+Furthermore, the model undergoes self-improvement by iteratively refining its
+training dataset. To verify the effectiveness of this approach, we conduct
+extensive experiments on three challenging benchmarks. The experimental results
+demonstrate that our approach achieves competitive performance compared to
+industry-level reasoning systems on these benchmarks.
+
+
+
+ comment: Technical Report on Slow Thinking with LLMs: Part II
+
+
+
+ + ☆ Neural Text Normalization for Luxembourgish using Real-Life Variation + Data +
+ +
+ Orthographic variation is very common in Luxembourgish texts due to the
+absence of a fully-fledged standard variety. Additionally, developing NLP tools
+for Luxembourgish is a difficult task given the lack of annotated and parallel
+data, which is exacerbated by ongoing standardization. In this paper, we
+propose the first sequence-to-sequence normalization models using the ByT5 and
+mT5 architectures with training data obtained from word-level real-life
+variation data. We perform a fine-grained, linguistically-motivated evaluation
+to test byte-based, word-based and pipeline-based models for their strengths
+and weaknesses in text normalization. We show that our sequence model using
+real-life variation data is an effective approach for tailor-made normalization
+in Luxembourgish.
+
+
+
+ comment: Accepted at VarDial 2025
+
+
+
+ + ☆ From Bench to Bedside: A Review of Clinical Trialsin Drug Discovery and + Development +
+
+
+
+
+
+
+
+ Tianyang Wang, Ming Liu, Benji Peng, Xinyuan Song, Charles Zhang, Xintian Sun, Qian Niu, Junyu Liu, Silin Chen, Keyu Chen, Ming Li, Pohsun Feng, Ziqian Bi, Yunze Wang, Yichao Zhang, Cheng Fei, Lawrence KQ Yan
+
+
+ Clinical trials are an indispensable part of the drug development process,
+bridging the gap between basic research and clinical application. During the
+development of new drugs, clinical trials are used not only to evaluate the
+safety and efficacy of the drug but also to explore its dosage, treatment
+regimens, and potential side effects. This review discusses the various stages
+of clinical trials, including Phase I (safety assessment), Phase II
+(preliminary efficacy evaluation), Phase III (large-scale validation), and
+Phase IV (post-marketing surveillance), highlighting the characteristics of
+each phase and their interrelationships. Additionally, the paper addresses the
+major challenges encountered in clinical trials, such as ethical issues,
+subject recruitment difficulties, diversity and representativeness concerns,
+and proposes strategies for overcoming these challenges. With the advancement
+of technology, innovative technologies such as artificial intelligence, big
+data, and digitalization are gradually transforming clinical trial design and
+implementation, improving trial efficiency and data quality. The article also
+looks forward to the future of clinical trials, particularly the impact of
+emerging therapies such as gene therapy and immunotherapy on trial design, as
+well as the importance of regulatory reforms and global collaboration. In
+conclusion, the core role of clinical trials in drug development will continue
+to drive the progress of innovative drug development and clinical treatment.
+
+
+
+ comment: 11 pages
+
+
+
+ + ☆ Word Sense Linking: Disambiguating Outside the Sandbox +
+
+
+
+
+
+
+
+ Andrei Stefan Bejgu, Edoardo Barba, Luigi Procopio, Alberte Fernández-Castro, Roberto Navigli
+
+
+ Word Sense Disambiguation (WSD) is the task of associating a word in a given
+context with its most suitable meaning among a set of possible candidates.
+While the task has recently witnessed renewed interest, with systems achieving
+performances above the estimated inter-annotator agreement, at the time of
+writing it still struggles to find downstream applications. We argue that one
+of the reasons behind this is the difficulty of applying WSD to plain text.
+Indeed, in the standard formulation, models work under the assumptions that a)
+all the spans to disambiguate have already been identified, and b) all the
+possible candidate senses of each span are provided, both of which are
+requirements that are far from trivial. In this work, we present a new task
+called Word Sense Linking (WSL) where, given an input text and a reference
+sense inventory, systems have to both identify which spans to disambiguate and
+then link them to their most suitable meaning.We put forward a
+transformer-based architecture for the task and thoroughly evaluate both its
+performance and those of state-of-the-art WSD systems scaled to WSL,
+iteratively relaxing the assumptions of WSD. We hope that our work will foster
+easier integration of lexical semantics into downstream applications.
+
+
+
+
+ + ☆ Falcon-UI: Understanding GUI Before Following User Instructions +
+ +
+ Pursuing human-like interaction for Graphical User Interface (GUI) agents
+requires understanding the GUI context and following user instructions.
+However, existing works typically couple these two aspects and focus more on
+instruct-following abilities, while ignoring the importance of understanding
+the GUI context. In this paper, we introduce an instruction-free GUI navigation
+dataset, termed Insight-UI Dataset, to enhance model comprehension of GUI
+environments. Insight-UI Dataset is automatically generated from the Common
+Crawl corpus, simulating various platforms -- including iOS, Android, Windows,
+and Linux -- across multiple resolutions on 312K domains. Although GUI
+interactions vary by context, diverse interfaces share common internal
+patterns, such as clicking an item to view its details. It implies the
+feasibility of independent GUI operation learning, followed by joint
+optimization with instruction tuning. Thereby, we develop the GUI agent model
+Falcon-UI, which is initially pretrained on Insight-UI Dataset and subsequently
+fine-tuned on Android and Web GUI datasets, including AITW, AITZ, Android
+Control, and Mind2Web. With 7 billion parameters, Falcon-UI achieves accuracy
+comparable to the 72 billion-parameter Qwen2VL on AITZ, validating the
+alignment between GUI context comprehension and agent performance. Our code and
+dataset will be open-sourced.
+
+
+
+ comment: 18 pages, 14 figures
+
+
+
+ + ☆ Causal Graphical Models for Vision-Language Compositional Understanding +
+
+
+
+
+
+
+
+ Fiorenzo Parascandolo, Nicholas Moratelli, Enver Sangineto, Lorenzo Baraldi, Rita Cucchiara
+
+
+ Recent work has empirically shown that Vision-Language Models (VLMs) struggle
+to fully understand the compositional properties of the human language, usually
+modeling an image caption as a "bag of words". As a result, they perform poorly
+on compositional tasks, which require a deeper understanding of the different
+entities of a sentence (subject, verb, etc.) jointly with their mutual
+relationships in order to be solved. In this paper, we model the dependency
+relations among textual and visual tokens using a Causal Graphical Model (CGM),
+built using a dependency parser, and we train a decoder conditioned by the VLM
+visual encoder. Differently from standard autoregressive or parallel
+predictions, our decoder's generative process is partially-ordered following
+the CGM structure. This structure encourages the decoder to learn only the main
+causal dependencies in a sentence discarding spurious correlations. Using
+extensive experiments on five compositional benchmarks, we show that our method
+significantly outperforms all the state-of-the-art compositional approaches by
+a large margin, and it also improves over methods trained using much larger
+datasets.
+
+
+
+
+ + ☆ Training LayoutLM from Scratch for Efficient Named-Entity Recognition in + the Insurance Domain +
+ +
+ Generic pre-trained neural networks may struggle to produce good results in
+specialized domains like finance and insurance. This is due to a domain
+mismatch between training data and downstream tasks, as in-domain data are
+often scarce due to privacy constraints. In this work, we compare different
+pre-training strategies for LayoutLM. We show that using domain-relevant
+documents improves results on a named-entity recognition (NER) problem using a
+novel dataset of anonymized insurance-related financial documents called
+Payslips. Moreover, we show that we can achieve competitive results using a
+smaller and faster model.
+
+
+
+ comment: Coling 2025 workshop (FinNLP)
+
+
+
+ + ☆ Benchmarking LLMs for Mimicking Child-Caregiver Language in Interaction +
+ +
+ LLMs can generate human-like dialogues, yet their ability to simulate early
+child-adult interactions remains largely unexplored. In this paper, we examined
+how effectively LLMs can capture the distinctive features of child-caregiver
+language in interaction, using both static and interactive benchmarking
+methods. We found that state-of-the-art LLMs like Llama 3 and GPT-4o can
+approximate child-caregiver dialogues at the word and utterance level, but they
+struggle to reproduce the child and caregiver's discursive patterns, exaggerate
+alignment, and fail to reach the level of diversity shown by humans. The
+broader goal of this work is to initiate the development of a comprehensive
+benchmark for LLMs in child-oriented applications.
+
+
+
+
+ + ☆ CRVQ: Channel-relaxed Vector Quantization for Extreme Compression of + LLMs +
+ +
+ Powerful large language models (LLMs) are increasingly expected to be
+deployed with lower computational costs, enabling their capabilities on
+resource-constrained devices. Post-training quantization (PTQ) has emerged as a
+star approach to achieve this ambition, with best methods compressing weights
+to less than 2 bit on average. In this paper, we propose Channel-Relaxed Vector
+Quantization (CRVQ), a novel technique that significantly improves the
+performance of PTQ baselines at the cost of only minimal additional bits. This
+state-of-the-art extreme compression method achieves its results through two
+key innovations: (1) carefully selecting and reordering a very small subset of
+critical weight channels, and (2) leveraging multiple codebooks to relax the
+constraint of critical channels. With our method, we demonstrate a 38.9%
+improvement over the current strongest sub-2-bit PTQ baseline, enabling nearer
+lossless 1-bit compression. Furthermore, our approach offers flexible
+customization of quantization bit-width and performance, providing a wider
+range of deployment options for diverse hardware platforms.
+
+
+
+ comment: 5 figures, 4 tables
+
+
+
+ + ☆ Learning to Solve Domain-Specific Calculation Problems with + Knowledge-Intensive Programs Generator +
+ +
+ Domain Large Language Models (LLMs) are developed for domain-specific tasks
+based on general LLMs. But it still requires professional knowledge to
+facilitate the expertise for some domain-specific tasks. In this paper, we
+investigate into knowledge-intensive calculation problems. We find that the
+math problems to be challenging for LLMs, when involving complex
+domain-specific rules and knowledge documents, rather than simple formulations
+of terminologies. Therefore, we propose a pipeline to solve the domain-specific
+calculation problems with Knowledge-Intensive Programs Generator more
+effectively, named as KIPG. It generates knowledge-intensive programs according
+to the domain-specific documents. For each query, key variables are extracted,
+then outcomes which are dependent on domain knowledge are calculated with the
+programs. By iterative preference alignment, the code generator learns to
+improve the logic consistency with the domain knowledge. Taking legal domain as
+an example, we have conducted experiments to prove the effectiveness of our
+pipeline, and extensive analysis on the modules. We also find that the code
+generator is also adaptable to other domains, without training on the new
+knowledge.
+
+
+
+ comment: Under review
+
+
+
+ + ☆ Towards Understanding the Robustness of LLM-based Evaluations under + Perturbations +
+ +
+ Traditional evaluation metrics like BLEU and ROUGE fall short when capturing
+the nuanced qualities of generated text, particularly when there is no single
+ground truth. In this paper, we explore the potential of Large Language Models
+(LLMs), specifically Google Gemini 1, to serve as automatic evaluators for
+non-standardized metrics in summarization and dialog-based tasks. We conduct
+experiments across multiple prompting strategies to examine how LLMs fare as
+quality evaluators when compared with human judgments on the SummEval and USR
+datasets, asking the model to generate both a score as well as a justification
+for the score. Furthermore, we explore the robustness of the LLM evaluator by
+using perturbed inputs. Our findings suggest that while LLMs show promise,
+their alignment with human evaluators is limited, they are not robust against
+perturbations and significant improvements are required for their standalone
+use as reliable evaluators for subjective metrics.
+
+
+
+ comment: Accepted at ICON 2024
+
+
+
+ + ☆ First Train to Generate, then Generate to Train: UnitedSynT5 for + Few-Shot NLI +
+ +
+ Natural Language Inference (NLI) tasks require identifying the relationship
+between sentence pairs, typically classified as entailment, contradiction, or
+neutrality. While the current state-of-the-art (SOTA) model, Entailment
+Few-Shot Learning (EFL), achieves a 93.1% accuracy on the Stanford Natural
+Language Inference (SNLI) dataset, further advancements are constrained by the
+dataset's limitations. To address this, we propose a novel approach leveraging
+synthetic data augmentation to enhance dataset diversity and complexity. We
+present UnitedSynT5, an advanced extension of EFL that leverages a T5-based
+generator to synthesize additional premise-hypothesis pairs, which are
+rigorously cleaned and integrated into the training data. These augmented
+examples are processed within the EFL framework, embedding labels directly into
+hypotheses for consistency. We train a GTR-T5-XL model on this expanded
+dataset, achieving a new benchmark of 94.7% accuracy on the SNLI dataset,
+94.01% accuracy on the E-SNLI dataset, and 92.57% accuracy on the MultiNLI
+dataset, surpassing the previous SOTA models. This research demonstrates the
+potential of synthetic data augmentation in improving NLI models, offering a
+path forward for further advancements in natural language understanding tasks.
+
+
+
+ comment: 14 pages
+
+
+
+ + ☆ Make Satire Boring Again: Reducing Stylistic Bias of Satirical Corpus by + Utilizing Generative LLMs COLING2025 +
+ +
+ Satire detection is essential for accurately extracting opinions from textual
+data and combating misinformation online. However, the lack of diverse corpora
+for satire leads to the problem of stylistic bias which impacts the models'
+detection performances. This study proposes a debiasing approach for satire
+detection, focusing on reducing biases in training data by utilizing generative
+large language models. The approach is evaluated in both cross-domain (irony
+detection) and cross-lingual (English) settings. Results show that the
+debiasing method enhances the robustness and generalizability of the models for
+satire and irony detection tasks in Turkish and English. However, its impact on
+causal language models, such as Llama-3.1, is limited. Additionally, this work
+curates and presents the Turkish Satirical News Dataset with detailed human
+annotations, with case studies on classification, debiasing, and
+explainability.
+
+
+
+ comment: Accepted to BUCC2025 Workshop @COLING2025
+
+
+
+ + ☆ CleanComedy: Creating Friendly Humor through Generative Techniques +
+
+
+
+
+
+
+
+ Dmitry Vikhorev, Daria Galimzianova, Svetlana Gorovaia, Elizaveta Zhemchuzhina, Ivan P. Yamshchikov
+
+
+ Humor generation is a challenging task in natural language processing due to
+limited resources and the quality of existing datasets. Available humor
+language resources often suffer from toxicity and duplication, limiting their
+effectiveness for training robust models. This paper proposes CleanComedy, a
+specialized, partially annotated toxicity-filtered corpus of English and
+Russian jokes collected from various sources. We study the effectiveness of our
+data filtering approach through a survey on humor and toxicity levels in
+various joke groups. In addition, we study advances in computer humor
+generation by comparing jokes written by humans with various groups of
+generative jokes, including our baseline models trained on the CleanComedy
+datasets.
+
+
+
+
+ + ☆ ReFF: Reinforcing Format Faithfulness in Language Models across Varied + Tasks AAAI 2025 +
+ +
+ Following formatting instructions to generate well-structured content is a
+fundamental yet often unmet capability for large language models (LLMs). To
+study this capability, which we refer to as format faithfulness, we present
+FormatBench, a comprehensive format-related benchmark. Compared to previous
+format-related benchmarks, FormatBench involves a greater variety of tasks in
+terms of application scenes (traditional NLP tasks, creative works, autonomous
+agency tasks), human-LLM interaction styles (single-turn instruction,
+multi-turn chat), and format types (inclusion, wrapping, length, coding).
+Moreover, each task in FormatBench is attached with a format checker program.
+Extensive experiments on the benchmark reveal that state-of-the-art open- and
+closed-source LLMs still suffer from severe deficiency in format faithfulness.
+By virtue of the decidable nature of formats, we propose to Reinforce Format
+Faithfulness (ReFF) to help LLMs generate formatted output as instructed
+without compromising general quality. Without any annotated data, ReFF can
+substantially improve the format faithfulness rate (e.g., from 21.6% in
+original LLaMA3 to 95.0% on caption segmentation task), while keep the general
+quality comparable (e.g., from 47.3 to 46.4 in F1 scores). Combined with
+labeled training data, ReFF can simultaneously improve both format faithfulness
+(e.g., from 21.6% in original LLaMA3 to 75.5%) and general quality (e.g., from
+47.3 to 61.6 in F1 scores). We further offer an interpretability analysis to
+explain how ReFF improves both format faithfulness and general quality.
+
+
+
+ comment: Accepted to AAAI 2025
+
+
+
+ + ☆ When Text Embedding Meets Large Language Model: A Comprehensive Survey +
+
+
+
+
+
+
+
+ Zhijie Nie, Zhangchi Feng, Mingxin Li, Cunwang Zhang, Yanzhao Zhang, Dingkun Long, Richong Zhang
+
+
+ Text embedding has become a foundational technology in natural language
+processing (NLP) during the deep learning era, driving advancements across a
+wide array of downstream tasks. While many natural language understanding
+challenges can now be modeled using generative paradigms and leverage the
+robust generative and comprehension capabilities of large language models
+(LLMs), numerous practical applications, such as semantic matching, clustering,
+and information retrieval, continue to rely on text embeddings for their
+efficiency and effectiveness. In this survey, we categorize the interplay
+between LLMs and text embeddings into three overarching themes: (1)
+LLM-augmented text embedding, enhancing traditional embedding methods with
+LLMs; (2) LLMs as text embedders, utilizing their innate capabilities for
+embedding generation; and (3) Text embedding understanding with LLMs,
+leveraging LLMs to analyze and interpret embeddings. By organizing these
+efforts based on interaction patterns rather than specific downstream
+applications, we offer a novel and systematic overview of contributions from
+various research and application domains in the era of LLMs. Furthermore, we
+highlight the unresolved challenges that persisted in the pre-LLM era with
+pre-trained language models (PLMs) and explore the emerging obstacles brought
+forth by LLMs. Building on this analysis, we outline prospective directions for
+the evolution of text embedding, addressing both theoretical and practical
+opportunities in the rapidly advancing landscape of NLP.
+
+
+
+ comment: Work in progress
+
+
+
+ + ☆ PolyIPA -- Multilingual Phoneme-to-Grapheme Conversion Model +
+ +
+ This paper presents PolyIPA, a novel multilingual phoneme-to-grapheme
+conversion model designed for multilingual name transliteration, onomastic
+research, and information retrieval. The model leverages two helper models
+developed for data augmentation: IPA2vec for finding soundalikes across
+languages, and similarIPA for handling phonetic notation variations. Evaluated
+on a test set that spans multiple languages and writing systems, the model
+achieves a mean Character Error Rate of 0.055 and a character-level BLEU score
+of 0.914, with particularly strong performance on languages with shallow
+orthographies. The implementation of beam search further improves practical
+utility, with top-3 candidates reducing the effective error rate by 52.7\% (to
+CER: 0.026), demonstrating the model's effectiveness for cross-linguistic
+applications.
+
+
+
+
+ + ☆ Filter-then-Generate: Large Language Models with Structure-Text Adapter + for Knowledge Graph Completion COLING 2025 +
+ +
+ Large Language Models (LLMs) present massive inherent knowledge and superior
+semantic comprehension capability, which have revolutionized various tasks in
+natural language processing. Despite their success, a critical gap remains in
+enabling LLMs to perform knowledge graph completion (KGC). Empirical evidence
+suggests that LLMs consistently perform worse than conventional KGC approaches,
+even through sophisticated prompt design or tailored instruction-tuning.
+Fundamentally, applying LLMs on KGC introduces several critical challenges,
+including a vast set of entity candidates, hallucination issue of LLMs, and
+under-exploitation of the graph structure. To address these challenges, we
+propose a novel instruction-tuning-based method, namely FtG. Specifically, we
+present a \textit{filter-then-generate} paradigm and formulate the KGC task
+into a multiple-choice question format. In this way, we can harness the
+capability of LLMs while mitigating the issue casused by hallucinations.
+Moreover, we devise a flexible ego-graph serialization prompt and employ a
+structure-text adapter to couple structure and text information in a
+contextualized manner. Experimental results demonstrate that FtG achieves
+substantial performance gain compared to existing state-of-the-art methods. The
+instruction dataset and code are available at
+\url{https://github.com/LB0828/FtG}.
+
+
+
+ comment: COLING 2025 Main Conference
+
+
+
+ + ☆ Evaluating Pixel Language Models on Non-Standardized Languages COLING 2025 +
+ +
+ We explore the potential of pixel-based models for transfer learning from
+standard languages to dialects. These models convert text into images that are
+divided into patches, enabling a continuous vocabulary representation that
+proves especially useful for out-of-vocabulary words common in dialectal data.
+Using German as a case study, we compare the performance of pixel-based models
+to token-based models across various syntactic and semantic tasks. Our results
+show that pixel-based models outperform token-based models in part-of-speech
+tagging, dependency parsing and intent detection for zero-shot dialect
+evaluation by up to 26 percentage points in some scenarios, though not in
+Standard German. However, pixel-based models fall short in topic
+classification. These findings emphasize the potential of pixel-based models
+for handling dialectal data, though further research should be conducted to
+assess their effectiveness in various linguistic contexts.
+
+
+
+ comment: Accepted at COLING 2025
+
+
+
+ + ☆ Forest-of-Thought: Scaling Test-Time Compute for Enhancing LLM Reasoning +
+ +
+ Large Language Models (LLMs) have shown remarkable abilities across various
+language tasks, but solving complex reasoning problems remains a challenge.
+While existing methods like Chain-of-Thought (CoT) and Tree-of-Thought (ToT)
+enhance reasoning by decomposing problems or structuring prompts, they
+typically perform a single pass of reasoning and may fail to revisit flawed
+paths, compromising accuracy. To address this, we propose a novel reasoning
+framework called Forest-of-Thought (FoT), which integrates multiple reasoning
+trees to leverage collective decision-making for solving complex logical
+problems. FoT utilizes sparse activation strategies to select the most relevant
+reasoning paths, improving both efficiency and accuracy. Additionally, we
+introduce a dynamic self-correction strategy that enables real-time error
+correction and learning from past mistakes, as well as consensus-guided
+decision making strategies to optimize correctness and computational resources.
+Experimental results demonstrate that the FoT framework, combined with these
+strategies, significantly enhances the reasoning capabilities of LLMs, enabling
+them to solve complex tasks with greater precision and efficiency.
+
+
+
+ comment: Preprint
+
+
+
+ + ☆ Dial-In LLM: Human-Aligned Dialogue Intent Clustering with + LLM-in-the-loop +
+ +
+ The discovery of customer intention from dialogue plays an important role in
+automated support system. However, traditional text clustering methods are
+poorly aligned with human perceptions due to the shift from embedding distance
+to semantic distance, and existing quantitative metrics for text clustering may
+not accurately reflect the true quality of intent clusters. In this paper, we
+leverage the superior language understanding capabilities of Large Language
+Models (LLMs) for designing better-calibrated intent clustering algorithms. We
+first establish the foundation by verifying the robustness of fine-tuned LLM
+utility in semantic coherence evaluation and cluster naming, resulting in an
+accuracy of 97.50% and 94.40%, respectively, when compared to the human-labeled
+ground truth. Then, we propose an iterative clustering algorithm that
+facilitates cluster-level refinement and the continuous discovery of
+high-quality intent clusters. Furthermore, we present several LLM-in-the-loop
+semi-supervised clustering techniques tailored for intent discovery from
+customer service dialogue. Experiments on a large-scale industrial dataset
+comprising 1,507 intent clusters demonstrate the effectiveness of the proposed
+techniques. The methods outperformed existing counterparts, achieving 6.25%
+improvement in quantitative metrics and 12% enhancement in application-level
+performance when constructing an intent classifier.
+
+
+
+
+ + ☆ Multi-Task Learning with LLMs for Implicit Sentiment Analysis: + Data-level and Task-level Automatic Weight Learning +
+ +
+ Implicit sentiment analysis (ISA) presents significant challenges due to the
+absence of salient cue words. Previous methods have struggled with insufficient
+data and limited reasoning capabilities to infer underlying opinions.
+Integrating multi-task learning (MTL) with large language models (LLMs) offers
+the potential to enable models of varying sizes to reliably perceive and
+recognize genuine opinions in ISA. However, existing MTL approaches are
+constrained by two sources of uncertainty: data-level uncertainty, arising from
+hallucination problems in LLM-generated contextual information, and task-level
+uncertainty, stemming from the varying capacities of models to process
+contextual information. To handle these uncertainties, we introduce MT-ISA, a
+novel MTL framework that enhances ISA by leveraging the generation and
+reasoning capabilities of LLMs through automatic MTL. Specifically, MT-ISA
+constructs auxiliary tasks using generative LLMs to supplement sentiment
+elements and incorporates automatic MTL to fully exploit auxiliary data. We
+introduce data-level and task-level automatic weight learning (AWL), which
+dynamically identifies relationships and prioritizes more reliable data and
+critical tasks, enabling models of varying sizes to adaptively learn
+fine-grained weights based on their reasoning capabilities. We investigate
+three strategies for data-level AWL, while also introducing homoscedastic
+uncertainty for task-level AWL. Extensive experiments reveal that models of
+varying sizes achieve an optimal balance between primary prediction and
+auxiliary tasks in MT-ISA. This underscores the effectiveness and adaptability
+of our approach.
+
+
+
+ comment: 11 pages, 6 figures, and 6 tables
+
+
+
+ + ☆ Mining Word Boundaries from Speech-Text Parallel Data for Cross-domain + Chinese Word Segmentation COLING 2025 +
+ +
+ Inspired by early research on exploring naturally annotated data for Chinese
+Word Segmentation (CWS), and also by recent research on integration of speech
+and text processing, this work for the first time proposes to explicitly mine
+word boundaries from speech-text parallel data. We employ the Montreal Forced
+Aligner (MFA) toolkit to perform character-level alignment on speech-text data,
+giving pauses as candidate word boundaries. Based on detailed analysis of
+collected pauses, we propose an effective probability-based strategy for
+filtering unreliable word boundaries. To more effectively utilize word
+boundaries as extra training data, we also propose a robust complete-then-train
+(CTT) strategy. We conduct cross-domain CWS experiments on two target domains,
+i.e., ZX and AISHELL2. We have annotated about 1,000 sentences as the
+evaluation data of AISHELL2. Experiments demonstrate the effectiveness of our
+proposed approach.
+
+
+
+ comment: COLING 2025
+
+
+
+ + ☆ ZigZagkv: Dynamic KV Cache Compression for Long-context Modeling based + on Layer Uncertainty +
+
+
+
+
+
+
+
+ Meizhi Zhong, Xikai Liu, Chen Zhang, Yikun Lei, Yan Gao, Yao Hu, Kehai Chen, Min Zhang
+
+
+ Large Language models (LLMs) have become a research hotspot. To accelerate
+the inference of LLMs, storing computed caches in memory has become the
+standard technique. However, as the inference length increases, growing KV
+caches might lead to out-of-memory issues. Many existing methods address this
+issue through KV cache compression, primarily by preserving key tokens
+throughout all layers to reduce information loss. Most of them allocate a
+uniform budget size for each layer to retain. However, we observe that the
+minimum budget sizes needed to retain essential information vary across layers
+and models based on the perspectives of attention and hidden state output.
+Building on this observation, this paper proposes a simple yet effective KV
+cache compression method that leverages layer uncertainty to allocate budget
+size for each layer. Experimental results show that the proposed method can
+reduce memory usage of the KV caches to only $\sim$20\% when compared to Full
+KV inference while achieving nearly lossless performance.
+
+
+
+
+ + ☆ Dialogue Language Model with Large-Scale Persona Data Engineering +
+ +
+ Maintaining persona consistency is paramount in the application of
+open-domain dialogue systems, as exemplified by models like ChatGPT. Despite
+significant advancements, the limited scale and diversity of current persona
+dialogue datasets remain challenges to achieving robust persona-consistent
+dialogue models. In this study, drawing inspiration from the success of
+large-scale pre-training, we introduce PPDS, an open-domain persona dialogue
+system that employs extensive generative pre-training on a persona dialogue
+dataset to enhance persona consistency. Specifically, we present a persona
+extraction model designed to autonomously and precisely generate vast persona
+dialogue datasets. Additionally, we unveil a pioneering persona augmentation
+technique to address the invalid persona bias inherent in the constructed
+dataset. Both quantitative and human evaluations consistently highlight the
+superior response quality and persona consistency of our proposed model,
+underscoring its effectiveness.
+
+
+
+
+ + ☆ Shiksha: A Technical Domain focused Translation Dataset and Model for + Indian Languages +
+ +
+ Neural Machine Translation (NMT) models are typically trained on datasets
+with limited exposure to Scientific, Technical and Educational domains.
+Translation models thus, in general, struggle with tasks that involve
+scientific understanding or technical jargon. Their performance is found to be
+even worse for low-resource Indian languages. Finding a translation dataset
+that tends to these domains in particular, poses a difficult challenge. In this
+paper, we address this by creating a multilingual parallel corpus containing
+more than 2.8 million rows of English-to-Indic and Indic-to-Indic high-quality
+translation pairs across 8 Indian languages. We achieve this by bitext mining
+human-translated transcriptions of NPTEL video lectures. We also finetune and
+evaluate NMT models using this corpus and surpass all other publicly available
+models at in-domain tasks. We also demonstrate the potential for generalizing
+to out-of-domain translation tasks by improving the baseline by over 2 BLEU on
+average for these Indian languages on the Flores+ benchmark. We are pleased to
+release our model and dataset via this link: https://huggingface.co/SPRINGLab.
+
+
+
+
+ + ☆ Improvement in Sign Language Translation Using Text CTC Alignment +
+ +
+ Current sign language translation (SLT) approaches often rely on gloss-based
+supervision with Connectionist Temporal Classification (CTC), limiting their
+ability to handle non-monotonic alignments between sign language video and
+spoken text. In this work, we propose a novel method combining joint
+CTC/Attention and transfer learning. The joint CTC/Attention introduces
+hierarchical encoding and integrates CTC with the attention mechanism during
+decoding, effectively managing both monotonic and non-monotonic alignments.
+Meanwhile, transfer learning helps bridge the modality gap between vision and
+language in SLT. Experimental results on two widely adopted benchmarks,
+RWTH-PHOENIX-Weather 2014 T and CSL-Daily, show that our method achieves
+results comparable to state-of-the-art and outperforms the pure-attention
+baseline. Additionally, this work opens a new door for future research into
+gloss-free SLT using text-based CTC alignment.
+
+
+
+
+ + ☆ What Makes Cryptic Crosswords Challenging for LLMs? COLING 2025 +
+ +
+ Cryptic crosswords are puzzles that rely on general knowledge and the
+solver's ability to manipulate language on different levels, dealing with
+various types of wordplay. Previous research suggests that solving such puzzles
+is challenging even for modern NLP models, including Large Language Models
+(LLMs). However, there is little to no research on the reasons for their poor
+performance on this task. In this paper, we establish the benchmark results for
+three popular LLMs: Gemma2, LLaMA3 and ChatGPT, showing that their performance
+on this task is still significantly below that of humans. We also investigate
+why these models struggle to achieve superior performance. We release our code
+and introduced datasets at
+https://github.com/bodasadallah/decrypting-crosswords.
+
+
+
+ comment: COLING 2025
+
+
+
+ + ☆ Assessing the Robustness of Retrieval-Augmented Generation Systems in + K-12 Educational Question Answering with Knowledge Discrepancies +
+ +
+ Retrieval-Augmented Generation (RAG) systems have demonstrated remarkable
+potential as question answering systems in the K-12 Education domain, where
+knowledge is typically queried within the restricted scope of authoritative
+textbooks. However, the discrepancy between textbooks and the parametric
+knowledge in Large Language Models (LLMs) could undermine the effectiveness of
+RAG systems. To systematically investigate the robustness of RAG systems under
+such knowledge discrepancies, we present EduKDQA, a question answering dataset
+that simulates knowledge discrepancies in real applications by applying
+hypothetical knowledge updates in answers and source documents. EduKDQA
+includes 3,005 questions covering five subjects, under a comprehensive question
+typology from the perspective of context utilization and knowledge integration.
+We conducted extensive experiments on retrieval and question answering
+performance. We find that most RAG systems suffer from a substantial
+performance drop in question answering with knowledge discrepancies, while
+questions that require integration of contextual knowledge and parametric
+knowledge pose a challenge to LLMs.
+
+
+
+ comment: 10 pages
+
+
+
+ + ☆ RuleArena: A Benchmark for Rule-Guided Reasoning with LLMs in Real-World + Scenarios +
+
+
+
+
+
+
+
+ Ruiwen Zhou, Wenyue Hua, Liangming Pan, Sitao Cheng, Xiaobao Wu, En Yu, William Yang Wang
+
+
+ This paper introduces RuleArena, a novel and challenging benchmark designed
+to evaluate the ability of large language models (LLMs) to follow complex,
+real-world rules in reasoning. Covering three practical domains -- airline
+baggage fees, NBA transactions, and tax regulations -- RuleArena assesses LLMs'
+proficiency in handling intricate natural language instructions that demand
+long-context understanding, logical reasoning, and accurate mathematical
+computation. Two key attributes distinguish RuleArena from traditional
+rule-based reasoning benchmarks: (1) it extends beyond standard first-order
+logic representations, and (2) it is grounded in authentic, practical
+scenarios, providing insights into the suitability and reliability of LLMs for
+real-world applications. Our findings reveal several notable limitations in
+LLMs: (1) they struggle to identify and apply the appropriate rules, frequently
+becoming confused by similar but distinct regulations, (2) they cannot
+consistently perform accurate mathematical computations, even when they
+correctly identify the relevant rules, and (3) in general, they perform poorly
+in the benchmark. These results highlight significant challenges in advancing
+LLMs' rule-guided reasoning capabilities in real-life applications.
+
+
+
+ comment: Data and Codes are available at
+ https://github.com/skyriver-2000/RuleArena
+
+
+
+ + ☆ Reasoning-Aware Query-Focused Summarization over Multi-Table Data +
+ +
+ Query-focused summarization over multi-table data is a challenging yet
+critical task for extracting precise and relevant information from structured
+data. Existing methods often rely on complex preprocessing steps and struggle
+to generalize across domains or handle the logical reasoning required for
+multi-table queries. In this paper, we propose QueryTableSummarizer++, an
+end-to-end generative framework leveraging large language models (LLMs)
+enhanced with table-aware pre-training, query-aligned fine-tuning, and
+reinforcement learning with feedback. Our method eliminates the need for
+intermediate serialization steps and directly generates query-relevant
+summaries. Experiments on a benchmark dataset demonstrate that
+QueryTableSummarizer++ significantly outperforms state-of-the-art baselines in
+terms of BLEU, ROUGE, and F1-score. Additional analyses highlight its
+scalability, generalization across domains, and robust handling of complex
+queries. Human evaluation further validates the superior quality and practical
+applicability of the generated summaries, establishing QueryTableSummarizer++
+as a highly effective solution for multi-table summarization tasks.
+
+
+
+
+ + ☆ Align, Generate, Learn: A Novel Closed-Loop Framework for Cross-Lingual + In-Context Learning +
+ +
+ Cross-lingual in-context learning (XICL) has emerged as a transformative
+paradigm for leveraging large language models (LLMs) to tackle multilingual
+tasks, especially for low-resource languages. However, existing approaches
+often rely on external retrievers or task-specific fine-tuning, limiting their
+scalability and generalizability. In this paper, we propose a novel
+self-supervised framework that harnesses the generative capabilities of LLMs to
+internally select and utilize task-relevant examples. Our method introduces two
+key objectives: a retrieval-generation alignment loss to optimize the quality
+of selected examples and a semantic coherence loss to ensure cross-lingual
+consistency. Through extensive experiments on multilingual benchmarks, our
+approach achieves state-of-the-art performance, significantly outperforming
+existing baselines. Further analysis highlights its robustness across diverse
+language families and its ability to generalize to unseen tasks. Human
+evaluations confirm the superior fluency, relevance, and semantic correctness
+of outputs generated by our method. This work provides a scalable, effective,
+and generalizable solution for cross-lingual in-context learning.
+
+
+
+
+ + ☆ Mojito: Motion Trajectory and Intensity Control for Video Generation +
+
+
+
+
+
+
+
+ Xuehai He, Shuohang Wang, Jianwei Yang, Xiaoxia Wu, Yiping Wang, Kuan Wang, Zheng Zhan, Olatunji Ruwase, Yelong Shen, Xin Eric Wang
+
+
+ Recent advancements in diffusion models have shown great promise in producing
+high-quality video content. However, efficiently training diffusion models
+capable of integrating directional guidance and controllable motion intensity
+remains a challenging and under-explored area. This paper introduces Mojito, a
+diffusion model that incorporates both \textbf{Mo}tion tra\textbf{j}ectory and
+\textbf{i}ntensi\textbf{t}y contr\textbf{o}l for text to video generation.
+Specifically, Mojito features a Directional Motion Control module that
+leverages cross-attention to efficiently direct the generated object's motion
+without additional training, alongside a Motion Intensity Modulator that uses
+optical flow maps generated from videos to guide varying levels of motion
+intensity. Extensive experiments demonstrate Mojito's effectiveness in
+achieving precise trajectory and intensity control with high computational
+efficiency, generating motion patterns that closely match specified directions
+and intensities, providing realistic dynamics that align well with natural
+motion in real-world scenarios.
+
+
+
+
+ + ☆ MoSLD: An Extremely Parameter-Efficient Mixture-of-Shared LoRAs for + Multi-Task Learning COLING 2025 +
+ +
+ Recently, LoRA has emerged as a crucial technique for fine-tuning large
+pre-trained models, yet its performance in multi-task learning scenarios often
+falls short. In contrast, the MoE architecture presents a natural solution to
+this issue. However, it introduces challenges such as mutual interference of
+data across multiple domains and knowledge forgetting of various tasks.
+Additionally, MoE significantly increases the number of parameters, posing a
+computational cost challenge. Therefore, in this paper, we propose MoSLD, a
+mixture-of-shared-LoRAs model with a dropout strategy. MoSLD addresses these
+challenges by sharing the upper projection matrix in LoRA among different
+experts, encouraging the model to learn general knowledge across tasks, while
+still allowing the lower projection matrix to focus on the unique features of
+each task. The application of dropout alleviates the imbalanced update of
+parameter matrix and mitigates parameter overfitting in LoRA. Extensive
+experiments demonstrate that our model exhibits excellent performance in both
+single-task and multi-task scenarios, with robust out-of-domain generalization
+capabilities.
+
+
+
+ comment: Accept by COLING 2025
+
+
+
+ + ☆ Multi-Scale Heterogeneous Text-Attributed Graph Datasets From Diverse + Domains +
+ +
+ Heterogeneous Text-Attributed Graphs (HTAGs), where different types of
+entities are not only associated with texts but also connected by diverse
+relationships, have gained widespread popularity and application across various
+domains. However, current research on text-attributed graph learning
+predominantly focuses on homogeneous graphs, which feature a single node and
+edge type, thus leaving a gap in understanding how methods perform on HTAGs.
+One crucial reason is the lack of comprehensive HTAG datasets that offer
+original textual content and span multiple domains of varying sizes. To this
+end, we introduce a collection of challenging and diverse benchmark datasets
+for realistic and reproducible evaluation of machine learning models on HTAGs.
+Our HTAG datasets are multi-scale, span years in duration, and cover a wide
+range of domains, including movie, community question answering, academic,
+literature, and patent networks. We further conduct benchmark experiments on
+these datasets with various graph neural networks. All source data, dataset
+construction codes, processed HTAGs, data loaders, benchmark codes, and
+evaluation setup are publicly available at GitHub and Hugging Face.
+
+
+
+
+ + ☆ From Text to Trajectory: Exploring Complex Constraint Representation and + Decomposition in Safe Reinforcement Learning NeurIPS 2024 +
+ +
+ Safe reinforcement learning (RL) requires the agent to finish a given task
+while obeying specific constraints. Giving constraints in natural language form
+has great potential for practical scenarios due to its flexible transfer
+capability and accessibility. Previous safe RL methods with natural language
+constraints typically need to design cost functions manually for each
+constraint, which requires domain expertise and lacks flexibility. In this
+paper, we harness the dual role of text in this task, using it not only to
+provide constraint but also as a training signal. We introduce the
+Trajectory-level Textual Constraints Translator (TTCT) to replace the manually
+designed cost function. Our empirical results demonstrate that TTCT effectively
+comprehends textual constraint and trajectory, and the policies trained by TTCT
+can achieve a lower violation rate than the standard cost function. Extra
+studies are conducted to demonstrate that the TTCT has zero-shot transfer
+capability to adapt to constraint-shift environments.
+
+
+
+ comment: Accepted by NeurIPS 2024
+
+
+
+ + ☆ Phi-4 Technical Report +
+
+
+
+
+
+
+
+ Marah Abdin, Jyoti Aneja, Harkirat Behl, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, Russell J. Hewett, Mojan Javaheripi, Piero Kauffmann, James R. Lee, Yin Tat Lee, Yuanzhi Li, Weishung Liu, Caio C. T. Mendes, Anh Nguyen, Eric Price, Gustavo de Rosa, Olli Saarikivi, Adil Salim, Shital Shah, Xin Wang, Rachel Ward, Yue Wu, Dingli Yu, Cyril Zhang, Yi Zhang
+
+
+ We present phi-4, a 14-billion parameter language model developed with a
+training recipe that is centrally focused on data quality. Unlike most language
+models, where pre-training is based primarily on organic data sources such as
+web content or code, phi-4 strategically incorporates synthetic data throughout
+the training process. While previous models in the Phi family largely distill
+the capabilities of a teacher model (specifically GPT-4), phi-4 substantially
+surpasses its teacher model on STEM-focused QA capabilities, giving evidence
+that our data-generation and post-training techniques go beyond distillation.
+Despite minimal changes to the phi-3 architecture, phi-4 achieves strong
+performance relative to its size -- especially on reasoning-focused benchmarks
+-- due to improved data, training curriculum, and innovations in the
+post-training scheme.
+
+
+
+
+ + ☆ AI-assisted Knowledge Discovery in Biomedical Literature to Support + Decision-making in Precision Oncology +
+
+
+
+
+
+
+
+ Ting He, Kory Kreimeyer, Mimi Najjar, Jonathan Spiker, Maria Fatteh, Valsamo Anagnostou, Taxiarchis Botsis
+
+
+ The delivery of appropriate targeted therapies to cancer patients requires
+the complete analysis of the molecular profiling of tumors and the patient's
+clinical characteristics in the context of existing knowledge and recent
+findings described in biomedical literature and several other sources. We
+evaluated the potential contributions of specific natural language processing
+solutions to support knowledge discovery from biomedical literature. Two models
+from the Bidirectional Encoder Representations from Transformers (BERT) family,
+two Large Language Models, and PubTator 3.0 were tested for their ability to
+support the named entity recognition (NER) and the relation extraction (RE)
+tasks. PubTator 3.0 and the BioBERT model performed best in the NER task (best
+F1-score equal to 0.93 and 0.89, respectively), while BioBERT outperformed all
+other solutions in the RE task (best F1-score 0.79) and a specific use case it
+was applied to by recognizing nearly all entity mentions and most of the
+relations.
+
+
+
+ comment: Accepted at AMIA Annual Symposium 2024
+
+
+
+ + ☆ A Graph-Based Synthetic Data Pipeline for Scaling High-Quality Reasoning + Instructions +
+
+
+
+
+
+
+
+ Jiankang Wang, Jianjun Xu, Xiaorui Wang, Yuxin Wang, Mengting Xing, Shancheng Fang, Zhineng Chen, Hongtao Xie, Yongdong Zhang
+
+
+ Synthesizing high-quality reasoning data for continual training has been
+proven to be effective in enhancing the performance of Large Language Models
+(LLMs). However, previous synthetic approaches struggle to easily scale up data
+and incur high costs in the pursuit of high quality. In this paper, we propose
+the Graph-based Synthetic Data Pipeline (GSDP), an economical and scalable
+framework for high-quality reasoning data synthesis. Inspired by knowledge
+graphs, we extracted knowledge points from seed data and constructed a
+knowledge point relationships graph to explore their interconnections. By
+exploring the implicit relationships among knowledge, our method achieves
+$\times$255 data expansion. Furthermore, GSDP led by open-source models,
+achieves synthesis quality comparable to GPT-4-0613 while maintaining
+$\times$100 lower costs. To tackle the most challenging mathematical reasoning
+task, we present the GSDP-MATH dataset comprising over 1.91 million pairs of
+math problems and answers. After fine-tuning on GSDP-MATH, GSDP-7B based on
+Mistral-7B achieves 37.7% accuracy on MATH and 78.4% on GSM8K, demonstrating
+the effectiveness of our method. The dataset and models trained in this paper
+will be available.
+
+
+
+
+ + ☆ Exploring Large Language Models on Cross-Cultural Values in Connection + with Training Methodology +
+ +
+ Large language models (LLMs) closely interact with humans, and thus need an
+intimate understanding of the cultural values of human society. In this paper,
+we explore how open-source LLMs make judgments on diverse categories of
+cultural values across countries, and its relation to training methodology such
+as model sizes, training corpus, alignment, etc. Our analysis shows that LLMs
+can judge socio-cultural norms similar to humans but less so on social systems
+and progress. In addition, LLMs tend to judge cultural values biased toward
+Western culture, which can be improved with training on the multilingual
+corpus. We also find that increasing model size helps a better understanding of
+social values, but smaller models can be enhanced by using synthetic data. Our
+analysis reveals valuable insights into the design methodology of LLMs in
+connection with their understanding of cultural values.
+
+
+
+
+ + ♻ ☆ Unveiling the Impact of Coding Data Instruction Fine-Tuning on Large + Language Models Reasoning +
+
+
+
+
+
+
+
+ Xinlu Zhang, Zhiyu Zoey Chen, Xi Ye, Xianjun Yang, Lichang Chen, William Yang Wang, Linda Ruth Petzold
+
+
+ Instruction Fine-Tuning (IFT) significantly enhances the zero-shot
+capabilities of pretrained Large Language Models (LLMs). While coding data is
+known to boost LLM reasoning abilities during pretraining, its role in
+activating internal reasoning capacities during IFT remains understudied. This
+paper investigates a key question: How does coding data impact LLMs' reasoning
+capacities during IFT stage? To explore this, we thoroughly examine the impact
+of coding data across different coding data proportions, model families, sizes,
+and reasoning domains, from various perspectives. Specifically, we create three
+IFT datasets with increasing coding data proportions, fine-tune six LLM
+backbones across different families and scales on these datasets, evaluate the
+tuned models' performance across twelve tasks in three reasoning domains, and
+analyze the outcomes from three broad-to-granular perspectives: overall,
+domain-level, and task-specific. Our holistic analysis provides valuable
+insights into each perspective. First, coding data tuning enhances the overall
+reasoning capabilities of LLMs across different model families and scales.
+Moreover, while the impact of coding data varies by domain, it shows consistent
+trends within each domain across different model families and scales.
+Additionally, coding data generally provides comparable task-specific benefits
+across model families, with optimal proportions in IFT datasets being
+task-dependent.
+
+
+
+
+ + ♻ ☆ Altogether: Image Captioning via Re-aligning Alt-text EMNLP 2024 +
+
+
+
+
+
+
+
+ Hu Xu, Po-Yao Huang, Xiaoqing Ellen Tan, Ching-Feng Yeh, Jacob Kahn, Christine Jou, Gargi Ghosh, Omer Levy, Luke Zettlemoyer, Wen-tau Yih, Shang-Wen Li, Saining Xie, Christoph Feichtenhofer
+
+
+ This paper focuses on creating synthetic data to improve the quality of image
+captions. Existing works typically have two shortcomings. First, they caption
+images from scratch, ignoring existing alt-text metadata, and second, lack
+transparency if the captioners' training data (e.g. GPT) is unknown. In this
+paper, we study a principled approach Altogether based on the key idea to edit
+and re-align existing alt-texts associated with the images. To generate
+training data, we perform human annotation where annotators start with the
+existing alt-text and re-align it to the image content in multiple rounds,
+consequently constructing captions with rich visual concepts. This differs from
+prior work that carries out human annotation as a one-time description task
+solely based on images and annotator knowledge. We train a captioner on this
+data that generalizes the process of re-aligning alt-texts at scale. Our
+results show our Altogether approach leads to richer image captions that also
+improve text-to-image generation and zero-shot image classification tasks.
+
+
+
+ comment: accepted by EMNLP 2024; Meta CLIP 1.2 Data Engine
+
+
+
+ + ♻ ☆ LCFO: Long Context and Long Form Output Dataset and Benchmarking +
+
+
+
+
+
+
+
+ Marta R. Costa-jussà, Pierre Andrews, Mariano Coria Meglioli, Joy Chen, Joe Chuang, David Dale, Christophe Ropers, Alexandre Mourachko, Eduardo Sánchez, Holger Schwenk, Tuan Tran, Arina Turkatenko, Carleigh Wood
+
+
+ This paper presents the Long Context and Form Output (LCFO) benchmark, a
+novel evaluation framework for assessing gradual summarization and summary
+expansion capabilities across diverse domains. LCFO consists of long input
+documents (5k words average length), each of which comes with three summaries
+of different lengths (20%, 10%, and 5% of the input text), as well as
+approximately 15 questions and answers (QA) related to the input content.
+Notably, LCFO also provides alignments between specific QA pairs and
+corresponding summaries in 7 domains. The primary motivation behind providing
+summaries of different lengths is to establish a controllable framework for
+generating long texts from shorter inputs, i.e. summary expansion. To establish
+an evaluation metric framework for summarization and summary expansion, we
+provide human evaluation scores for human-generated outputs, as well as results
+from various state-of-the-art large language models (LLMs). GPT-4o-mini
+achieves best human scores among automatic systems in both summarization and
+summary expansion tasks (~ +10% and +20%, respectively). It even surpasses
+human output quality in the case of short summaries (~ +7%). Overall automatic
+metrics achieve low correlations with human evaluation scores (~ 0.4) but
+moderate correlation on specific evaluation aspects such as fluency and
+attribution (~ 0.6). The LCFO benchmark offers a standardized platform for
+evaluating summarization and summary expansion performance, as well as
+corresponding automatic metrics, thereby providing an important evaluation
+framework to advance generative AI.
+
+
+
+
+ + ♻ ☆ Improving the Validity of Automatically Generated Feedback via + Reinforcement Learning +
+ +
+ Automatically generating feedback via large language models (LLMs) in
+intelligent tutoring systems and online learning platforms has the potential to
+improve the learning outcomes of many students. However, both feedback
+generation and evaluation are challenging: feedback content has to be valid
+especially in subjects like math, which requires models to understand the
+problem, the solution, and where the student's error lies. Feedback also has to
+be pedagogically valid to reflect effective tutoring strategies, such as
+explaining possible misconceptions and encouraging the student, among other
+desirable features. In this work, we address both problems of automatically
+generating and evaluating feedback while considering both correctness and
+alignment. First, we propose a rubric for evaluating math feedback and show
+that GPT-4 is able to effectively use it to annotate human-written and
+LLM-generated feedback. Second, we propose a framework for feedback generation
+that optimizes both correctness and alignment using reinforcement learning
+(RL). Specifically, we use GPT-4's annotations to create preferences over
+feedback pairs in an augmented dataset for training via direct preference
+optimization (DPO). We show that our methods significantly increase the
+correctness and alignment of generated feedback with Llama 2, an open-source
+LLM, qualitatively analyze our generation and evaluation systems using case
+studies, and outline several areas for future work.
+
+
+
+ comment: Best student paper award, Published in AIED 2024: The 25th
+ International Conference on Artificial Intelligence in Education
+
+
+
+ + ♻ ☆ Evaluating GPT-4 at Grading Handwritten Solutions in Math Exams +
+ +
+ Recent advances in generative artificial intelligence (AI) have shown promise
+in accurately grading open-ended student responses. However, few prior works
+have explored grading handwritten responses due to a lack of data and the
+challenge of combining visual and textual information. In this work, we
+leverage state-of-the-art multi-modal AI models, in particular GPT-4o, to
+automatically grade handwritten responses to college-level math exams. Using
+real student responses to questions in a probability theory exam, we evaluate
+GPT-4o's alignment with ground-truth scores from human graders using various
+prompting techniques. We find that while providing rubrics improves alignment,
+the model's overall accuracy is still too low for real-world settings, showing
+there is significant room for growth in this task.
+
+
+
+ comment: Published in LAK 2025: The 15th International Learning Analytics and
+ Knowledge Conference
+
+
+
+ + ♻ ☆ SAGED: A Holistic Bias-Benchmarking Pipeline for Language Models with + Customisable Fairness Calibration COLING 2025 +
+
+
+
+
+
+
+
+ Xin Guan, Nathaniel Demchak, Saloni Gupta, Ze Wang, Ediz Ertekin Jr., Adriano Koshiyama, Emre Kazim, Zekun Wu
+
+
+ The development of unbiased large language models is widely recognized as
+crucial, yet existing benchmarks fall short in detecting biases due to limited
+scope, contamination, and lack of a fairness baseline. SAGED(bias) is the first
+holistic benchmarking pipeline to address these problems. The pipeline
+encompasses five core stages: scraping materials, assembling benchmarks,
+generating responses, extracting numeric features, and diagnosing with
+disparity metrics. SAGED includes metrics for max disparity, such as impact
+ratio, and bias concentration, such as Max Z-scores. Noticing that metric tool
+bias and contextual bias in prompts can distort evaluation, SAGED implements
+counterfactual branching and baseline calibration for mitigation. For
+demonstration, we use SAGED on G20 Countries with popular 8b-level models
+including Gemma2, Llama3.1, Mistral, and Qwen2. With sentiment analysis, we
+find that while Mistral and Qwen2 show lower max disparity and higher bias
+concentration than Gemma2 and Llama3.1, all models are notably biased against
+countries like Russia and (except for Qwen2) China. With further experiments to
+have models role-playing U.S. presidents, we see bias amplifies and shifts in
+heterogeneous directions. Moreover, we see Qwen2 and Mistral not engage in
+role-playing, while Llama3.1 and Gemma2 role-play Trump notably more
+intensively than Biden and Harris, indicating role-playing performance bias in
+these models.
+
+
+
+ comment: COLING 2025 Main Conference
+
+
+
+ + ♻ ☆ Few-Shot Domain Adaptation for Named-Entity Recognition via Joint + Constrained k-Means and Subspace Selection COLING 2025 +
+ +
+ Named-entity recognition (NER) is a task that typically requires large
+annotated datasets, which limits its applicability across domains with varying
+entity definitions. This paper addresses few-shot NER, aiming to transfer
+knowledge to new domains with minimal supervision. Unlike previous approaches
+that rely solely on limited annotated data, we propose a weakly supervised
+algorithm that combines small labeled datasets with large amounts of unlabeled
+data. Our method extends the k-means algorithm with label supervision, cluster
+size constraints and domain-specific discriminative subspace selection. This
+unified framework achieves state-of-the-art results in few-shot NER on several
+English datasets.
+
+
+
+ comment: COLING 2025
+
+
+
+ + ♻ ☆ Dspy-based Neural-Symbolic Pipeline to Enhance Spatial Reasoning in LLMs +
+ +
+ Large Language Models (LLMs) have demonstrated remarkable capabilities across
+various tasks, yet they often struggle with spatial reasoning. This paper
+presents a novel neural-symbolic framework that enhances LLMs' spatial
+reasoning abilities through iterative feedback between LLMs and Answer Set
+Programming (ASP). We evaluate our approach on two benchmark datasets: StepGame
+and SparQA, implementing three distinct strategies: (1) direct prompting
+baseline, (2) Facts+Rules prompting, and (3) DSPy-based LLM+ASP pipeline with
+iterative refinement. Our experimental results demonstrate that the LLM+ASP
+pipeline significantly outperforms baseline methods, achieving an average 82%
+accuracy on StepGame and 69% on SparQA, marking improvements of 40-50% and
+8-15% respectively over direct prompting. The success stems from three key
+innovations: (1) effective separation of semantic parsing and logical reasoning
+through a modular pipeline, (2) iterative feedback mechanism between LLMs and
+ASP solvers that improves program rate, and (3) robust error handling that
+addresses parsing, grounding, and solving failures. Additionally, we propose
+Facts+Rules as a lightweight alternative that achieves comparable performance
+on complex SparQA dataset, while reducing computational overhead.Our analysis
+across different LLM architectures (Deepseek, Llama3-70B, GPT-4.0 mini)
+demonstrates the framework's generalizability and provides insights into the
+trade-offs between implementation complexity and reasoning capability,
+contributing to the development of more interpretable and reliable AI systems.
+
+
+
+
+ + ♻ ☆ EVQAScore: Efficient Video Question Answering Data Evaluation +
+ +
+ Video question-answering (QA) is a core task in video understanding.
+Evaluating the quality of video QA and video caption data quality for training
+video large language models (VideoLLMs) is an essential challenge. Although
+various methods have been proposed for assessing video caption quality, there
+remains a lack of dedicated evaluation methods for Video QA. To address this
+gap, we introduce EVQAScore, a reference-free method that leverages keyword
+extraction to assess both video caption and video QA data quality.
+Additionally, we incorporate frame sampling and rescaling techniques to enhance
+the efficiency and robustness of our evaluation, this enables our score to
+evaluate the quality of extremely long videos. Our approach achieves
+state-of-the-art (SOTA) performance (32.8 for Kendall correlation and 42.3 for
+Spearman correlation, 4.7 and 5.9 higher than the previous method PAC-S++) on
+the VATEX-EVAL benchmark for video caption evaluation. Furthermore, by using
+EVQAScore for data selection, we achieved SOTA results with only 12.5\% of the
+original data volume, outperforming the previous SOTA method PAC-S and 100\% of
+data.
+
+
+
+
+ + ♻ ☆ Detection of Non-recorded Word Senses in English and Swedish +
+ +
+ This study addresses the task of Unknown Sense Detection in English and
+Swedish. The primary objective of this task is to determine whether the meaning
+of a particular word usage is documented in a dictionary or not. For this
+purpose, sense entries are compared with word usages from modern and historical
+corpora using a pre-trained Word-in-Context embedder that allows us to model
+this task in a few-shot scenario. Additionally, we use human annotations on the
+target corpora to adapt hyperparameters and evaluate our models using 5-fold
+cross-validation. Compared to a random sample from a corpus, our model is able
+to considerably increase the detected number of word usages with non-recorded
+senses.
+
+
+
+ comment: 9 pages
+
+
+
+ + ♻ ☆ Importance Weighting Can Help Large Language Models Self-Improve +
+ +
+ Large language models (LLMs) have shown remarkable capability in numerous
+tasks and applications. However, fine-tuning LLMs using high-quality datasets
+under external supervision remains prohibitively expensive. In response, LLM
+self-improvement approaches have been vibrantly developed recently. The typical
+paradigm of LLM self-improvement involves training LLM on self-generated data,
+part of which may be detrimental and should be filtered out due to the unstable
+data quality. While current works primarily employs filtering strategies based
+on answer correctness, in this paper, we demonstrate that filtering out correct
+but with high distribution shift extent (DSE) samples could also benefit the
+results of self-improvement. Given that the actual sample distribution is
+usually inaccessible, we propose a new metric called DS weight to approximate
+DSE, inspired by the Importance Weighting methods. Consequently, we integrate
+DS weight with self-consistency to comprehensively filter the self-generated
+samples and fine-tune the language model. Experiments show that with only a
+tiny valid set (up to 5\% size of the training set) to compute DS weight, our
+approach can notably promote the reasoning ability of current LLM
+self-improvement methods. The resulting performance is on par with methods that
+rely on external supervision from pre-trained reward models.
+
+
+
+
+ + ♻ ☆ Exploring Language Model Generalization in Low-Resource Extractive QA COLING 2025 +
+ +
+ In this paper, we investigate Extractive Question Answering (EQA) with Large
+Language Models (LLMs) under domain drift, i.e., can LLMs generalize to domains
+that require specific knowledge such as medicine and law in a zero-shot fashion
+without additional in-domain training? To this end, we devise a series of
+experiments to explain the performance gap empirically. Our findings suggest
+that: (a) LLMs struggle with dataset demands of closed domains such as
+retrieving long answer spans; (b) Certain LLMs, despite showing strong overall
+performance, display weaknesses in meeting basic requirements as discriminating
+between domain-specific senses of words which we link to pre-processing
+decisions; (c) Scaling model parameters is not always effective for cross
+domain generalization; and (d) Closed-domain datasets are quantitatively much
+different than open-domain EQA datasets and current LLMs struggle to deal with
+them. Our findings point out important directions for improving existing LLMs.
+
+
+
+ comment: Accepted to COLING 2025
+
+
+
+ + ♻ ☆ TOP-Training: Target-Oriented Pretraining for Medical Extractive + Question Answering COLING 2025 +
+
+
+
+
+
+
+
+ Saptarshi Sengupta, Connor Heaton, Shreya Ghosh, Wenpeng Yin, Preslav Nakov, Suhang Wang
+
+
+ We study extractive question-answering in the medical domain (Medical-EQA).
+This problem has two main challenges: (i) domain specificity, as most AI models
+lack necessary domain knowledge, and (ii) extraction-based answering style,
+which restricts most autoregressive LLMs due to potential hallucinations. To
+handle those challenges, we propose TOP-Training, a target-oriented
+pre-training paradigm that stands out among all domain adaptation techniques
+with two desirable features: (i) TOP-Training moves one step further than
+popular domain-oriented fine-tuning since it not only moves closer to the
+target domain, but also familiarizes itself with the target dataset, and (ii)
+it does not assume the existence of a large set of unlabeled instances from the
+target domain. Specifically, for a target Medical-EQA dataset, we extract its
+entities and leverage large language models (LLMs) to generate synthetic texts
+containing those entities; we then demonstrate that pretraining on this
+synthetic text data yields better performance on the target Medical-EQA
+benchmarks. Overall, our contributions are threefold: (i) TOP-Training, a new
+pretraining technique to effectively adapt LLMs to better solve a target
+problem, (ii) TOP-Training has a wide application scope because it does not
+require the target problem to have a large set of unlabeled data, and (iii) our
+experiments highlight the limitations of autoregressive LLMs, emphasizing
+TOP-Training as a means to unlock the true potential of bidirectional LLMs.
+
+
+
+ comment: Accepted to COLING 2025
+
+
+
+ + ♻ ☆ Empathy Level Alignment via Reinforcement Learning for Empathetic + Response Generation +
+ +
+ Empathetic response generation, aiming to understand the user's situation and
+feelings and respond empathically, is crucial in building human-like dialogue
+systems. Traditional approaches typically employ maximum likelihood estimation
+as the optimization objective during training, yet fail to align the empathy
+levels between generated and target responses. To this end, we propose an
+empathetic response generation framework using reinforcement learning (EmpRL).
+The framework develops an effective empathy reward function and generates
+empathetic responses by maximizing the expected reward through reinforcement
+learning. EmpRL utilizes the pre-trained T5 model as the generator and further
+fine-tunes it to initialize the policy. To align the empathy levels between
+generated and target responses within a given context, an empathy reward
+function containing three empathy communication mechanisms -- emotional
+reaction, interpretation, and exploration -- is constructed using pre-designed
+and pre-trained empathy identifiers. During reinforcement learning training,
+the proximal policy optimization algorithm is used to fine-tune the policy,
+enabling the generation of empathetic responses. Both automatic and human
+evaluations demonstrate that the proposed EmpRL framework significantly
+improves the quality of generated responses, enhances the similarity in empathy
+levels between generated and target responses, and produces empathetic
+responses covering both affective and cognitive aspects.
+
+
+
+
+ + ♻ ☆ How Likely Do LLMs with CoT Mimic Human Reasoning? COLING 2025 +
+ +
+ Chain-of-thought emerges as a promising technique for eliciting reasoning
+capabilities from Large Language Models (LLMs). However, it does not always
+improve task performance or accurately represent reasoning processes, leaving
+unresolved questions about its usage. In this paper, we diagnose the underlying
+mechanism by comparing the reasoning process of LLMs with humans, using causal
+analysis to understand the relationships between the problem instruction,
+reasoning, and the answer in LLMs. Our empirical study reveals that LLMs often
+deviate from the ideal causal chain, resulting in spurious correlations and
+potential consistency errors (inconsistent reasoning and answers). We also
+examine various factors influencing the causal structure, finding that
+in-context learning with examples strengthens it, while post-training
+techniques like supervised fine-tuning and reinforcement learning on human
+feedback weaken it. To our surprise, the causal structure cannot be
+strengthened by enlarging the model size only, urging research on new
+techniques. We hope that this preliminary study will shed light on
+understanding and improving the reasoning process in LLM.
+
+
+
+ comment: COLING 2025 Camera Version (8 pages, 3 figures, 18 tables)
+
+
+
+ + ♻ ☆ NLPineers@ NLU of Devanagari Script Languages 2025: Hate Speech + Detection using Ensembling of BERT-based models +
+ +
+ This paper explores hate speech detection in Devanagari-scripted languages,
+focusing on Hindi and Nepali, for Subtask B of the CHIPSAL@COLING 2025 Shared
+Task. Using a range of transformer-based models such as XLM-RoBERTa, MURIL, and
+IndicBERT, we examine their effectiveness in navigating the nuanced boundary
+between hate speech and free expression. Our best performing model, implemented
+as ensemble of multilingual BERT models achieve Recall of 0.7762 (Rank 3/31 in
+terms of recall) and F1 score of 0.6914 (Rank 17/31). To address class
+imbalance, we used backtranslation for data augmentation, and cosine similarity
+to preserve label consistency after augmentation. This work emphasizes the need
+for hate speech detection in Devanagari-scripted languages and presents a
+foundation for further research.
+
+
+
+
+ + ♻ ☆ ProSparse: Introducing and Enhancing Intrinsic Activation Sparsity + within Large Language Models +
+
+
+
+
+
+
+
+ Chenyang Song, Xu Han, Zhengyan Zhang, Shengding Hu, Xiyu Shi, Kuai Li, Chen Chen, Zhiyuan Liu, Guangli Li, Tao Yang, Maosong Sun
+
+
+ Activation sparsity refers to the existence of considerable
+weakly-contributed elements among activation outputs. As a prevalent property
+of the models using the ReLU activation function, activation sparsity has been
+proven a promising paradigm to boost model inference efficiency. Nevertheless,
+most large language models (LLMs) adopt activation functions without intrinsic
+activation sparsity (e.g., GELU and Swish). Some recent efforts have explored
+introducing ReLU or its variants as the substitutive activation function to
+help LLMs achieve activation sparsity and inference acceleration, but few can
+simultaneously obtain high sparsity and comparable model performance. This
+paper introduces a simple and effective sparsification method named "ProSparse"
+to push LLMs for higher activation sparsity while maintaining comparable
+performance. Specifically, after substituting the activation function of LLMs
+with ReLU, ProSparse adopts progressive sparsity regularization with a factor
+smoothly increasing along the multi-stage sine curves. This can enhance
+activation sparsity and mitigate performance degradation by avoiding radical
+shifts in activation distributions. With ProSparse, we obtain high sparsity of
+89.32% for LLaMA2-7B, 88.80% for LLaMA2-13B, and 87.89% for end-size
+MiniCPM-1B, respectively, achieving comparable performance to their original
+Swish-activated versions. These present the most sparsely activated models
+among open-source LLaMA versions and competitive end-size models, considerably
+surpassing ReluLLaMA-7B (66.98%) and ReluLLaMA-13B (71.56%). Our inference
+acceleration experiments further demonstrate the significant practical
+acceleration potential of LLMs with higher activation sparsity, obtaining up to
+4.52$\times$ inference speedup.
+
+
+
+ comment: 19 pages, 4 figures, 9 tables
+
+
+
+ + ♻ ☆ Missing Melodies: AI Music Generation and its "Nearly" Complete Omission + of the Global South +
+ +
+ Recent advances in generative AI have sparked renewed interest and expanded
+possibilities for music generation. However, the performance and versatility of
+these systems across musical genres are heavily influenced by the availability
+of training data. We conducted an extensive analysis of over one million hours
+of audio datasets used in AI music generation research and manually reviewed
+more than 200 papers from eleven prominent AI and music conferences and
+organizations (AAAI, ACM, EUSIPCO, EURASIP, ICASSP, ICML, IJCAI, ISMIR,
+NeurIPS, NIME, SMC) to identify a critical gap in the fair representation and
+inclusion of the musical genres of the Global South in AI research. Our
+findings reveal a stark imbalance: approximately 86% of the total dataset hours
+and over 93% of researchers focus primarily on music from the Global North.
+However, around 40% of these datasets include some form of non-Western music,
+genres from the Global South account for only 14.6% of the data. Furthermore,
+approximately 51% of the papers surveyed concentrate on symbolic music
+generation, a method that often fails to capture the cultural nuances inherent
+in music from regions such as South Asia, the Middle East, and Africa. As AI
+increasingly shapes the creation and dissemination of music, the significant
+underrepresentation of music genres in datasets and research presents a serious
+threat to global musical diversity. We also propose some important steps to
+mitigate these risks and foster a more inclusive future for AI-driven music
+generation.
+
+
+
+ comment: Submitted to CACM, 12 pages, 2 figures
+
+
+
+ + ♻ ☆ Large language models as oracles for instantiating ontologies with + domain-specific knowledge +
+ +
+ Background. Endowing intelligent systems with semantic data commonly requires
+designing and instantiating ontologies with domain-specific knowledge.
+Especially in the early phases, those activities are typically performed
+manually by human experts possibly leveraging on their own experience. The
+resulting process is therefore time-consuming, error-prone, and often biased by
+the personal background of the ontology designer. Objective. To mitigate that
+issue, we propose a novel domain-independent approach to automatically
+instantiate ontologies with domain-specific knowledge, by leveraging on large
+language models (LLMs) as oracles. Method. Starting from (i) an initial schema
+composed by inter-related classes and properties and (ii) a set of query
+templates, our method queries the LLM multiple times, and generates instances
+for both classes and properties from its replies. Thus, the ontology is
+automatically filled with domain-specific knowledge, compliant to the initial
+schema. As a result, the ontology is quickly and automatically enriched with
+manifold instances, which experts may consider to keep, adjust, discard, or
+complement according to their own needs and expertise. Contribution. We
+formalise our method in general way and instantiate it over various LLMs, as
+well as on a concrete case study. We report experiments rooted in the
+nutritional domain where an ontology of food meals and their ingredients is
+automatically instantiated from scratch, starting from a categorisation of
+meals and their relationships. There, we analyse the quality of the generated
+ontologies and compare ontologies attained by exploiting different LLMs.
+Experimentally, our approach achieves a quality metric that is up to five times
+higher than the state-of-the-art, while reducing erroneous entities and
+relations by up to ten times. Finally, we provide a SWOT analysis of the
+proposed method.
+
+
+
+
+ + ♻ ☆ UniBias: Unveiling and Mitigating LLM Bias through Internal Attention + and FFN Manipulation NeurIPS 2024 +
+ +
+ Large language models (LLMs) have demonstrated impressive capabilities in
+various tasks using the in-context learning (ICL) paradigm. However, their
+effectiveness is often compromised by inherent bias, leading to prompt
+brittleness, i.e., sensitivity to design settings such as example selection,
+order, and prompt formatting. Previous studies have addressed LLM bias through
+external adjustment of model outputs, but the internal mechanisms that lead to
+such bias remain unexplored. Our work delves into these mechanisms,
+particularly investigating how feedforward neural networks (FFNs) and attention
+heads result in the bias of LLMs. By Interpreting the contribution of
+individual FFN vectors and attention heads, we identify the biased LLM
+components that skew LLMs' prediction toward specific labels. To mitigate these
+biases, we introduce UniBias, an inference-only method that effectively
+identifies and eliminates biased FFN vectors and attention heads. Extensive
+experiments across 12 NLP datasets demonstrate that UniBias significantly
+enhances ICL performance and alleviates prompt brittleness of LLMs.
+
+
+
+ comment: Accepted to NeurIPS 2024
+
+
+
+ + ♻ ☆ TCM-FTP: Fine-Tuning Large Language Models for Herbal Prescription + Prediction +
+
+
+
+
+
+
+
+ Xingzhi Zhou, Xin Dong, Chunhao Li, Yuning Bai, Yulong Xu, Ka Chun Cheung, Simon See, Xinpeng Song, Runshun Zhang, Xuezhong Zhou, Nevin L. Zhang
+
+
+ Traditional Chinese medicine (TCM) has relied on specific combinations of
+herbs in prescriptions to treat various symptoms and signs for thousands of
+years. Predicting TCM prescriptions poses a fascinating technical challenge
+with significant practical implications. However, this task faces limitations
+due to the scarcity of high-quality clinical datasets and the complex
+relationship between symptoms and herbs. To address these issues, we introduce
+\textit{DigestDS}, a novel dataset comprising practical medical records from
+experienced experts in digestive system diseases. We also propose a method,
+TCM-FTP (TCM Fine-Tuning Pre-trained), to leverage pre-trained large language
+models (LLMs) via supervised fine-tuning on \textit{DigestDS}. Additionally, we
+enhance computational efficiency using a low-rank adaptation technique.
+Moreover, TCM-FTP incorporates data augmentation by permuting herbs within
+prescriptions, exploiting their order-agnostic nature. Impressively, TCM-FTP
+achieves an F1-score of 0.8031, significantly outperforming previous methods.
+Furthermore, it demonstrates remarkable accuracy in dosage prediction,
+achieving a normalized mean square error of 0.0604. In contrast, LLMs without
+fine-tuning exhibit poor performance. Although LLMs have demonstrated
+wide-ranging capabilities, our work underscores the necessity of fine-tuning
+for TCM prescription prediction and presents an effective way to accomplish
+this.
+
+
+
+ comment: Camera-ready version to be published in BIBM 2024
+
+
+
+ + ♻ ☆ TouchTTS: An Embarrassingly Simple TTS Framework that Everyone Can Touch +
+
+
+
+
+
+
+
+ Xingchen Song, Mengtao Xing, Changwei Ma, Shengqiang Li, Di Wu, Binbin Zhang, Fuping Pan, Dinghao Zhou, Yuekai Zhang, Shun Lei, Zhendong Peng, Zhiyong Wu
+
+
+ It is well known that LLM-based systems are data-hungry. Recent LLM-based TTS
+works typically employ complex data processing pipelines to obtain high-quality
+training data. These sophisticated pipelines require excellent models at each
+stage (e.g., speech denoising, speech enhancement, speaker diarization, and
+punctuation models), which themselves demand high-quality training data and are
+rarely open-sourced. Even with state-of-the-art models, issues persist, such as
+incomplete background noise removal and misalignment between punctuation and
+actual speech pauses. Moreover, the stringent filtering strategies often retain
+only 10-30\% of the original data, significantly impeding data scaling efforts.
+In this work, we leverage a noise-robust audio tokenizer (S3Tokenizer) to
+design a simplified yet effective TTS data processing pipeline that maintains
+data quality while substantially reducing data acquisition costs, achieving a
+data retention rate of over 50\%. Beyond data scaling challenges, LLM-based TTS
+systems also incur higher deployment costs compared to conventional approaches.
+Current systems typically use LLMs solely for text-to-token generation, while
+requiring separate models (e.g., flow matching models) for token-to-waveform
+generation, which cannot be directly executed by LLM inference engines, further
+complicating deployment. To address these challenges, we eliminate redundant
+modules in both LLM and flow components, replacing the flow model backbone with
+an LLM architecture. Building upon this simplified flow backbone, we propose a
+unified architecture for both streaming and non-streaming inference,
+significantly reducing deployment costs. Finally, we explore the feasibility of
+unifying TTS and ASR tasks using the same data for training, thanks to the
+simplified pipeline and the S3Tokenizer that reduces the quality requirements
+for TTS training data.
+
+
+
+ comment: Technical Report
+
+
+
+ + ♻ ☆ ProSwitch: Knowledge-Guided Instruction Tuning to Switch Between + Professional and Non-Professional Responses +
+
+
+
+
+
+
+
+ Chang Zong, Yuyan Chen, Weiming Lu, Jian Shao, Yongfeng Huang, Heng Chang, Yueting Zhuang
+
+
+ Large Language Models (LLMs) have demonstrated efficacy in various linguistic
+applications, including question answering and controlled text generation.
+However, studies into their ability to switch between opposite styles of
+responses in professional domains remain underexplored. This study introduces a
+novel approach, named ProSwitch, which enables a language model to switch
+between professional and non-professional answers, by tuning and evaluating
+through the guidance of domain and style knowledge. ProSwitch unfolds in three
+phases: LLM-augmented preparation to collect domain knowledge and QA pairs,
+instruction tuning to optimize LLMs with multiple levels of knowledge, and
+comprehensive evaluation to assess both style discrimination and
+reference-based quality of the generated text. Comparative analysis of
+ProSwitch against general and specialized LLMs reveals that our approach
+outperforms baselines in switching between professional and non-professional
+responses.
+
+
+
+ comment: 8 pages main body, 16 pages total
+
+
+
+ + ♻ ☆ Evolving Alignment via Asymmetric Self-Play +
+
+
+
+
+
+
+
+ Ziyu Ye, Rishabh Agarwal, Tianqi Liu, Rishabh Joshi, Sarmishta Velury, Quoc V. Le, Qijun Tan, Yuan Liu
+
+
+ Current RLHF frameworks for aligning large language models (LLMs) typically
+assume a fixed prompt distribution, which is sub-optimal and limits the
+scalability of alignment and generalizability of models. To address this, we
+introduce a general open-ended RLHF framework that casts alignment as an
+asymmetric game between two players: (i) a creator that generates increasingly
+informative prompt distributions using reward signals, and (ii) a solver that
+learns to produce more preferred responses on prompts produced by the creator.
+This framework of Evolving Alignment via Asymmetric Self-Play (eva), results in
+a simple and efficient approach that can utilize any existing RLHF algorithm
+for scalable alignment. eva outperforms state-of-the-art methods on widely-used
+benchmarks, without the need of any additional human crafted prompts.
+Specifically, eva improves the win rate of Gemma-2-9B-it on Arena-Hard from
+51.6% to 60.1% with DPO, from 55.7% to 58.9% with SPPO, from 52.3% to 60.7%
+with SimPO, and from 54.8% to 60.3% with ORPO, surpassing its 27B version and
+matching claude-3-opus. This improvement is persistent even when new human
+crafted prompts are introduced. Finally, we show eva is effective and robust
+under various ablation settings.
+
+
+
+ comment: 35 pages, spotlight @ neurips language gamification workshop
+
+
+
+ + ♻ ☆ Match, Compare, or Select? An Investigation of Large Language Models for + Entity Matching COLING 2025 +
+
+
+
+
+
+
+
+ Tianshu Wang, Xiaoyang Chen, Hongyu Lin, Xuanang Chen, Xianpei Han, Hao Wang, Zhenyu Zeng, Le Sun
+
+
+ Entity matching (EM) is a critical step in entity resolution (ER). Recently,
+entity matching based on large language models (LLMs) has shown great promise.
+However, current LLM-based entity matching approaches typically follow a binary
+matching paradigm that ignores the global consistency among record
+relationships. In this paper, we investigate various methodologies for
+LLM-based entity matching that incorporate record interactions from different
+perspectives. Specifically, we comprehensively compare three representative
+strategies: matching, comparing, and selecting, and analyze their respective
+advantages and challenges in diverse scenarios. Based on our findings, we
+further design a compound entity matching framework (ComEM) that leverages the
+composition of multiple strategies and LLMs. ComEM benefits from the advantages
+of different sides and achieves improvements in both effectiveness and
+efficiency. Experimental results on 8 ER datasets and 10 LLMs verify the
+superiority of incorporating record interactions through the selecting
+strategy, as well as the further cost-effectiveness brought by ComEM.
+
+
+
+ comment: Accepted at COLING 2025. Our code is available at
+ https://github.com/tshu-w/ComEM
+
+
+
+ + ♻ ☆ Understanding the RoPE Extensions of Long-Context LLMs: An Attention + Perspective +
+
+
+
+
+
+
+
+ Meizhi Zhong, Chen Zhang, Yikun Lei, Xikai Liu, Yan Gao, Yao Hu, Kehai Chen, Min Zhang
+
+
+ Enabling LLMs to handle lengthy context is currently a research hotspot. Most
+LLMs are built upon rotary position embedding (RoPE), a popular position
+encoding method. Therefore, a prominent path is to extrapolate the RoPE trained
+on comparably short texts to far longer texts. A heavy bunch of efforts have
+been dedicated to boosting the extrapolation via extending the formulations of
+the RoPE, however, few of them have attempted to showcase their inner workings
+comprehensively. In this paper, we are driven to offer a straightforward yet
+in-depth understanding of RoPE extensions from an attention perspective and on
+two benchmarking tasks. A broad array of experiments reveals several valuable
+findings: 1) Maintaining attention patterns to those at the pretrained length
+improves extrapolation; 2) Large attention uncertainty leads to retrieval
+errors; 3) Using longer continual pretraining lengths for RoPE extensions could
+reduce attention uncertainty and significantly enhance extrapolation.
+
+
+
+
+ + ♻ ☆ Reasoning Graph Enhanced Exemplars Retrieval for In-Context Learning +
+ +
+ Large language models (LLMs) have exhibited remarkable few-shot learning
+capabilities and unified the paradigm of NLP tasks through the in-context
+learning (ICL) technique. Despite the success of ICL, the quality of the
+exemplar demonstrations can significantly influence the LLM's performance.
+Existing exemplar selection methods mainly focus on the semantic similarity
+between queries and candidate exemplars. On the other hand, the logical
+connections between reasoning steps can be beneficial to depict the
+problem-solving process as well. In this paper, we proposes a novel method
+named Reasoning Graph-enhanced Exemplar Retrieval (RGER). RGER first quires LLM
+to generate an initial response, then expresses intermediate problem-solving
+steps to a graph structure. After that, it employs graph kernel to select
+exemplars with semantic and structural similarity. Extensive experiments
+demonstrate the structural relationship is helpful to the alignment of queries
+and candidate exemplars. The efficacy of RGER on math and logit reasoning tasks
+showcases its superiority over state-of-the-art retrieval-based approaches. Our
+code is released at https://github.com/Yukang-Lin/RGER.
+
+
+
+
+ + ♻ ☆ Training on the Test Task Confounds Evaluation and Emergence +
+ +
+ We study a fundamental problem in the evaluation of large language models
+that we call training on the test task. Unlike wrongful practices like training
+on the test data, leakage, or data contamination, training on the test task is
+not a malpractice. Rather, the term describes a growing set of practices that
+utilize knowledge about evaluation tasks at training time. We demonstrate that
+training on the test task confounds both relative model evaluations and claims
+about emergent capabilities. We argue that the seeming superiority of one model
+family over another may be explained by a different degree of training on the
+test task. To this end, we propose an effective method to adjust for the effect
+of training on the test task on benchmark evaluations. Put simply, to fine-tune
+each model under comparison on the same task-relevant data before evaluation.
+We then show that instances of emergent behavior disappear gradually as models
+train on the test task. Our work promotes a new perspective on the evaluation
+of large language models with broad implications for benchmarking and the study
+of emergent capabilities
+
+
+
+
+ + ♻ ☆ Deep Learning and Machine Learning, Advancing Big Data Analytics and + Management: Unveiling AI's Potential Through Tools, Techniques, and + Applications +
+
+
+
+
+
+
+
+ Pohsun Feng, Ziqian Bi, Yizhu Wen, Xuanhe Pan, Benji Peng, Ming Liu, Jiawei Xu, Keyu Chen, Junyu Liu, Caitlyn Heqi Yin, Sen Zhang, Jinlang Wang, Qian Niu, Ming Li, Tianyang Wang
+
+
+ Artificial intelligence (AI), machine learning, and deep learning have become
+transformative forces in big data analytics and management, enabling
+groundbreaking advancements across diverse industries. This article delves into
+the foundational concepts and cutting-edge developments in these fields, with a
+particular focus on large language models (LLMs) and their role in natural
+language processing, multimodal reasoning, and autonomous decision-making.
+Highlighting tools such as ChatGPT, Claude, and Gemini, the discussion explores
+their applications in data analysis, model design, and optimization.
+ The integration of advanced algorithms like neural networks, reinforcement
+learning, and generative models has enhanced the capabilities of AI systems to
+process, visualize, and interpret complex datasets. Additionally, the emergence
+of technologies like edge computing and automated machine learning (AutoML)
+democratizes access to AI, empowering users across skill levels to engage with
+intelligent systems. This work also underscores the importance of ethical
+considerations, transparency, and fairness in the deployment of AI
+technologies, paving the way for responsible innovation.
+ Through practical insights into hardware configurations, software
+environments, and real-world applications, this article serves as a
+comprehensive resource for researchers and practitioners. By bridging
+theoretical underpinnings with actionable strategies, it showcases the
+potential of AI and LLMs to revolutionize big data management and drive
+meaningful advancements across domains such as healthcare, finance, and
+autonomous systems.
+
+
+
+ comment: This book contains 155 pages and 9 figures
+
+
+
+ + ♻ ☆ Controlled Evaluation of Syntactic Knowledge in Multilingual Language + Models COLING 2025 +
+ +
+ Language models (LMs) are capable of acquiring elements of human-like
+syntactic knowledge. Targeted syntactic evaluation tests have been employed to
+measure how well they form generalizations about syntactic phenomena in
+high-resource languages such as English. However, we still lack a thorough
+understanding of LMs' capacity for syntactic generalizations in low-resource
+languages, which are responsible for much of the diversity of syntactic
+patterns worldwide. In this study, we develop targeted syntactic evaluation
+tests for three low-resource languages (Basque, Hindi, and Swahili) and use
+them to evaluate five families of open-access multilingual Transformer LMs. We
+find that some syntactic tasks prove relatively easy for LMs while others
+(agreement in sentences containing indirect objects in Basque, agreement across
+a prepositional phrase in Swahili) are challenging. We additionally uncover
+issues with publicly available Transformers, including a bias toward the
+habitual aspect in Hindi in multilingual BERT and underperformance compared to
+similar-sized models in XGLM-4.5B.
+
+
+
+ comment: LoResLM workshop at COLING 2025
+
+
+
+ + ♻ ☆ Exploring the Limitations of Detecting Machine-Generated Text COLING 2025 +
+
+
+
+
+
+
+
+ Jad Doughman, Osama Mohammed Afzal, Hawau Olamide Toyin, Shady Shehata, Preslav Nakov, Zeerak Talat
+
+
+ Recent improvements in the quality of the generations by large language
+models have spurred research into identifying machine-generated text. Such work
+often presents high-performing detectors. However, humans and machines can
+produce text in different styles and domains, yet the performance impact of
+such on machine generated text detection systems remains unclear. In this
+paper, we audit the classification performance for detecting machine-generated
+text by evaluating on texts with varying writing styles. We find that
+classifiers are highly sensitive to stylistic changes and differences in text
+complexity, and in some cases degrade entirely to random classifiers. We
+further find that detection systems are particularly susceptible to misclassify
+easy-to-read texts while they have high performance for complex texts, leading
+to concerns about the reliability of detection systems. We recommend that
+future work attends to stylistic factors and reading difficulty levels of
+human-written and machine-generated text.
+
+
+
+ comment: Accepted to COLING 2025
+
+
+
+ + ♻ ☆ Guiding Vision-Language Model Selection for Visual Question-Answering + Across Tasks, Domains, and Knowledge Types COLING +
+ +
+ Visual Question-Answering (VQA) has become key to user experience,
+particularly after improved generalization capabilities of Vision-Language
+Models (VLMs). But evaluating VLMs for an application requirement using a
+standardized framework in practical settings is still challenging. This paper
+aims to solve that using an end-to-end framework. We present VQA360 - a novel
+dataset derived from established VQA benchmarks, annotated with task types,
+application domains, and knowledge types, for a comprehensive evaluation. We
+also introduce GoEval, a multimodal evaluation metric developed using GPT-4o,
+achieving a correlation factor of 56.71% with human judgments. Our experiments
+with state-of-the-art VLMs reveal that no single model excels universally,
+thus, making a right choice a key design decision. Proprietary models such as
+Gemini-1.5-Pro and GPT-4o-mini generally outperform others, but open-source
+models like InternVL-2-8B and CogVLM-2-Llama-3-19B also demonstrate competitive
+strengths, while providing additional advantages. Our framework can also be
+extended to other tasks.
+
+
+
+ comment: Accepted at The First Workshop of Evaluation of Multi-Modal
+ Generation (EvalMG) in 31st International Conference on Computational
+ Linguistics (COLING), 2025. 8 pages + references + 6 pages of Appendix
+
+
+
+ + ♻ ☆ Rumor Detection on Social Media with Temporal Propagation Structure + Optimization COLING'25 +
+ +
+ Traditional methods for detecting rumors on social media primarily focus on
+analyzing textual content, often struggling to capture the complexity of online
+interactions. Recent research has shifted towards leveraging graph neural
+networks to model the hierarchical conversation structure that emerges during
+rumor propagation. However, these methods tend to overlook the temporal aspect
+of rumor propagation and may disregard potential noise within the propagation
+structure. In this paper, we propose a novel approach that incorporates
+temporal information by constructing a weighted propagation tree, where the
+weight of each edge represents the time interval between connected posts.
+Drawing upon the theory of structural entropy, we transform this tree into a
+coding tree. This transformation aims to preserve the essential structure of
+rumor propagation while reducing noise. Finally, we introduce a recursive
+neural network to learn from the coding tree for rumor veracity prediction.
+Experimental results on two common datasets demonstrate the superiority of our
+approach.
+
+
+
+ comment: COLING'25
+
+
+
+ + ♻ ☆ VickreyFeedback: Cost-efficient Data Construction for Reinforcement + Learning from Human Feedback +
+ +
+ This paper addresses the cost-efficiency aspect of Reinforcement Learning
+from Human Feedback (RLHF). RLHF leverages datasets of human preferences over
+outputs of large language models (LLM)s to instill human expectations into
+LLMs. Although preference annotation comes with a monetized cost, the economic
+utility of a preference dataset has not been considered by far. What
+exacerbates this situation is that, given complex intransitive or cyclic
+relationships in preference datasets, existing algorithms for fine-tuning LLMs
+are still far from capturing comprehensive preferences. This raises severe
+cost-efficiency concerns in production environments, where preference data
+accumulate over time. In this paper, we discuss the fine-tuning of LLMs as a
+monetized economy and introduce an auction mechanism to improve the efficiency
+of preference data collection in dollar terms. We show that introducing an
+auction mechanism can play an essential role in enhancing the cost-efficiency
+of RLHF, while maintaining satisfactory model performance. Experimental results
+demonstrate that our proposed auction-based protocol is cost-effective for
+fine-tuning LLMs concentrating on high-quality feedback.
+
+
+
+ comment: 16 pages, 5 figures
+
+
+
+ + ♻ ☆ Return of EM: Entity-driven Answer Set Expansion for QA Evaluation COLING 2025 +
+ +
+ Recently, directly using large language models (LLMs) has been shown to be
+the most reliable method to evaluate QA models. However, it suffers from
+limited interpretability, high cost, and environmental harm. To address these,
+we propose to use soft EM with entity-driven answer set expansion. Our approach
+expands the gold answer set to include diverse surface forms, based on the
+observation that the surface forms often follow particular patterns depending
+on the entity type. The experimental results show that our method outperforms
+traditional evaluation methods by a large margin. Moreover, the reliability of
+our evaluation method is comparable to that of LLM-based ones, while offering
+the benefits of high interpretability and reduced environmental harm.
+
+
+
+ comment: Accepted at COLING 2025 (16 pages, 4 figures, 11 tables)
+
+
+
+ + ♻ ☆ Learn To be Efficient: Build Structured Sparsity in Large Language + Models +
+
+
+
+
+
+
+
+ Haizhong Zheng, Xiaoyan Bai, Xueshen Liu, Z. Morley Mao, Beidi Chen, Fan Lai, Atul Prakash
+
+
+ Large Language Models (LLMs) have achieved remarkable success with their
+billion-level parameters, yet they incur high inference overheads. The
+emergence of activation sparsity in LLMs provides a natural approach to reduce
+this cost by involving only parts of the parameters for inference. However,
+existing methods only focus on utilizing this naturally formed activation
+sparsity in a post-training setting, overlooking the potential for further
+amplifying this inherent sparsity. In this paper, we hypothesize that LLMs can
+learn to be efficient by achieving more structured activation sparsity. To
+achieve this, we introduce a novel training algorithm, Learn-To-be-Efficient
+(LTE), designed to train efficiency-aware LLMs to learn to activate fewer
+neurons and achieve a better trade-off between sparsity and performance.
+Furthermore, unlike SOTA MoEfication methods, which mainly focus on ReLU-based
+models, LTE can also be applied to LLMs like LLaMA using non-ReLU activations.
+Extensive evaluation on language understanding, language generation, and
+instruction tuning tasks show that LTE consistently outperforms SOTA baselines.
+Along with our hardware-aware custom kernel implementation, LTE reduces
+LLaMA2-7B inference latency by 25% at 50% sparsity.
+
+
+
+
+ + ♻ ☆ Universal Prompt Optimizer for Safe Text-to-Image Generation +
+ +
+ Text-to-Image (T2I) models have shown great performance in generating images
+based on textual prompts. However, these models are vulnerable to unsafe input
+to generate unsafe content like sexual, harassment and illegal-activity images.
+Existing studies based on image checker, model fine-tuning and embedding
+blocking are impractical in real-world applications. Hence, we propose the
+first universal prompt optimizer for safe T2I (POSI) generation in black-box
+scenario. We first construct a dataset consisting of toxic-clean prompt pairs
+by GPT-3.5 Turbo. To guide the optimizer to have the ability of converting
+toxic prompt to clean prompt while preserving semantic information, we design a
+novel reward function measuring toxicity and text alignment of generated images
+and train the optimizer through Proximal Policy Optimization. Experiments show
+that our approach can effectively reduce the likelihood of various T2I models
+in generating inappropriate images, with no significant impact on text
+alignment. It is also flexible to be combined with methods to achieve better
+performance. Our code is available at https://github.com/wu-zongyu/POSI.
+
+
+
+
+ + ♻ ☆ Adaptive Prompting for Continual Relation Extraction: A Within-Task + Variance Perspective AAAI 2025 +
+
+
+
+
+
+
+
+ Minh Le, Tien Ngoc Luu, An Nguyen The, Thanh-Thien Le, Trang Nguyen, Tung Thanh Nguyen, Linh Ngo Van, Thien Huu Nguyen
+
+
+ To address catastrophic forgetting in Continual Relation Extraction (CRE),
+many current approaches rely on memory buffers to rehearse previously learned
+knowledge while acquiring new tasks. Recently, prompt-based methods have
+emerged as potent alternatives to rehearsal-based strategies, demonstrating
+strong empirical performance. However, upon analyzing existing prompt-based
+approaches for CRE, we identified several critical limitations, such as
+inaccurate prompt selection, inadequate mechanisms for mitigating forgetting in
+shared parameters, and suboptimal handling of cross-task and within-task
+variances. To overcome these challenges, we draw inspiration from the
+relationship between prefix-tuning and mixture of experts, proposing a novel
+approach that employs a prompt pool for each task, capturing variations within
+each task while enhancing cross-task variances. Furthermore, we incorporate a
+generative model to consolidate prior knowledge within shared parameters,
+eliminating the need for explicit data storage. Extensive experiments validate
+the efficacy of our approach, demonstrating superior performance over
+state-of-the-art prompt-based and rehearsal-free methods in continual relation
+extraction.
+
+
+
+ comment: Accepted to AAAI 2025
+
+
+
+ + ♻ ☆ Trustful LLMs: Customizing and Grounding Text Generation with Knowledge + Bases and Dual Decoders +
+ +
+ Although people are impressed by the content generation skills of large
+language models, the use of LLMs, such as ChatGPT, is limited by the domain
+grounding of the content. The correctness and groundedness of the generated
+content need to be based on a verified context, such as results from
+Retrieval-Augmented Generation (RAG). One important issue when adapting LLMs to
+a customized domain is that the generated responses are often incomplete, or
+the additions are not verified and may even be hallucinated. Prior studies on
+hallucination detection have focused on evaluation metrics, which are not
+easily adaptable to dynamic domains and can be vulnerable to attacks like
+jail-breaking. In this work, we propose 1) a post-processing algorithm that
+leverages knowledge triplets in RAG context to correct hallucinations and 2) a
+dual-decoder model that fuses RAG context to guide the generation process.
+
+
+
+
+ + ♻ ☆ Frame Representation Hypothesis: Multi-Token LLM Interpretability and + Concept-Guided Text Generation +
+ +
+ Interpretability is a key challenge in fostering trust for Large Language
+Models (LLMs), which stems from the complexity of extracting reasoning from
+model's parameters. We present the Frame Representation Hypothesis, a
+theoretically robust framework grounded in the Linear Representation Hypothesis
+(LRH) to interpret and control LLMs by modeling multi-token words. Prior
+research explored LRH to connect LLM representations with linguistic concepts,
+but was limited to single token analysis. As most words are composed of several
+tokens, we extend LRH to multi-token words, thereby enabling usage on any
+textual data with thousands of concepts. To this end, we propose words can be
+interpreted as frames, ordered sequences of vectors that better capture
+token-word relationships. Then, concepts can be represented as the average of
+word frames sharing a common concept. We showcase these tools through Top-k
+Concept-Guided Decoding, which can intuitively steer text generation using
+concepts of choice. We verify said ideas on Llama 3.1, Gemma 2, and Phi 3
+families, demonstrating gender and language biases, exposing harmful content,
+but also potential to remediate them, leading to safer and more transparent
+LLMs. Code is available at
+https://github.com/phvv-me/frame-representation-hypothesis.git
+
+
+
+
+ + ♻ ☆ If You Can't Use Them, Recycle Them: Optimizing Merging at Scale + Mitigates Performance Tradeoffs +
+
+
+
+
+
+
+
+ Muhammad Khalifa, Yi-Chern Tan, Arash Ahmadian, Tom Hosking, Honglak Lee, Lu Wang, Ahmet Üstün, Tom Sherborne, Matthias Gallé
+
+
+ Model merging has shown great promise at combining expert models, but the
+benefit of merging is unclear when merging ``generalist'' models trained on
+many tasks. We explore merging in the context of large (~100B) models, by
+recycling checkpoints that exhibit tradeoffs among different tasks. Such
+checkpoints are often created in the process of developing a frontier model,
+and many suboptimal ones are usually discarded. Given a pool of model
+checkpoints obtained from different training runs (e.g., different stages,
+objectives, hyperparameters, and data mixtures), which naturally show tradeoffs
+across different language capabilities (e.g., instruction following vs. code
+generation), we investigate whether merging can recycle such suboptimal models
+into a Pareto-optimal one. Our optimization algorithm tunes the weight of each
+checkpoint in a linear combination, resulting in a Pareto-optimal models that
+outperforms both individual models and merge-based baselines. Further analysis
+shows that good merges tend to include almost all checkpoints with non-zero
+weights, indicating that even seemingly bad initial checkpoints can contribute
+to good final merges.
+
+
+
+ comment: 13 pages, 9 figures
+
+
+
+ + ♻ ☆ Revolutionizing Finance with LLMs: An Overview of Applications and + Insights +
+
+
+
+
+
+
+
+ Huaqin Zhao, Zhengliang Liu, Zihao Wu, Yiwei Li, Tianze Yang, Peng Shu, Shaochen Xu, Haixing Dai, Lin Zhao, Hanqi Jiang, Yi Pan, Junhao Chen, Yifan Zhou, Gengchen Mai, Ninghao Liu, Tianming Liu
+
+
+ In recent years, Large Language Models (LLMs) like ChatGPT have seen
+considerable advancements and have been applied in diverse fields. Built on the
+Transformer architecture, these models are trained on extensive datasets,
+enabling them to understand and generate human language effectively. In the
+financial domain, the deployment of LLMs is gaining momentum. These models are
+being utilized for automating financial report generation, forecasting market
+trends, analyzing investor sentiment, and offering personalized financial
+advice. Leveraging their natural language processing capabilities, LLMs can
+distill key insights from vast financial data, aiding institutions in making
+informed investment choices and enhancing both operational efficiency and
+customer satisfaction. In this study, we provide a comprehensive overview of
+the emerging integration of LLMs into various financial tasks. Additionally, we
+conducted holistic tests on multiple financial tasks through the combination of
+natural language instructions. Our findings show that GPT-4 effectively follow
+prompt instructions across various financial tasks. This survey and evaluation
+of LLMs in the financial domain aim to deepen the understanding of LLMs'
+current role in finance for both financial practitioners and LLM researchers,
+identify new research and application prospects, and highlight how these
+technologies can be leveraged to solve practical challenges in the finance
+industry.
+
+
+
+
+ + ♻ ☆ AI-Press: A Multi-Agent News Generating and Feedback Simulation System + Powered by Large Language Models +
+
+
+
+
+
+
+
+ Xiawei Liu, Shiyue Yang, Xinnong Zhang, Haoyu Kuang, Libo Sun, Yihang Yang, Siming Chen, Xuanjing Huang, Zhongyu Wei
+
+
+ The rise of various social platforms has transformed journalism. The growing
+demand for news content has led to the increased use of large language models
+(LLMs) in news production due to their speed and cost-effectiveness. However,
+LLMs still encounter limitations in professionalism and ethical judgment in
+news generation. Additionally, predicting public feedback is usually difficult
+before news is released. To tackle these challenges, we introduce AI-Press, an
+automated news drafting and polishing system based on multi-agent collaboration
+and Retrieval-Augmented Generation. We develop a feedback simulation system
+that generates public feedback considering demographic distributions. Through
+extensive quantitative and qualitative evaluations, our system shows
+significant improvements in news-generating capabilities and verifies the
+effectiveness of public feedback simulation.
+
+
+
+ comment: 18 pages, 4 figures
+
+
+
+ + ♻ ☆ Counting-Stars: A Multi-evidence, Position-aware, and Scalable Benchmark + for Evaluating Long-Context Large Language Models COLING 2025 +
+ +
+ Despite recent efforts to develop large language models with robust
+long-context capabilities, the lack of long-context benchmarks means that
+relatively little is known about their performance. To alleviate this gap, in
+this paper, we propose \textbf{Counting-Stars}, a multi-evidence,
+position-aware, and scalable benchmark designed to evaluate the multi-evidence
+retrieval capabilities of long-context LLMs. \textbf{Counting-Stars} comprises
+two counting-based multiple pieces of evidence retrieval tasks: searching and
+reasoning. Using Counting-Stars, we conducted experiments to evaluate several
+long-context LLMs, including GPT-4 Turbo, Gemini 1.5 Pro, Claude3 Opus, GLM-4,
+and Moonshot-v1. Extensive experimental results demonstrate that Gemini 1.5 Pro
+achieves the best overall results, while GPT-4 Turbo exhibits the most stable
+performance across various tasks. Furthermore, our analysis of these LLMs,
+which have been extended to handle long-context scenarios, indicates that
+significant room for improvement remains as the length of the input context and
+the complexity of the tasks increase.
+
+
+
+ comment: Accepted by COLING 2025
+
+
+
+ + ♻ ☆ Hierarchical Prompting Taxonomy: A Universal Evaluation Framework for + Large Language Models Aligned with Human Cognitive Principles +
+
+
+
+
+
+
+
+ Devichand Budagam, Ashutosh Kumar, Mahsa Khoshnoodi, Sankalp KJ, Vinija Jain, Aman Chadha
+
+
+ Assessing the effectiveness of large language models (LLMs) in performing
+different tasks is crucial for understanding their strengths and weaknesses.
+This paper presents Hierarchical Prompting Taxonomy (HPT), grounded on human
+cognitive principles and designed to assess LLMs by examining the cognitive
+demands of various tasks. The HPT utilizes the Hierarchical Prompting Framework
+(HPF), which structures five unique prompting strategies in a hierarchical
+order based on their cognitive requirement on LLMs when compared to human
+mental capabilities. It assesses the complexity of tasks with the Hierarchical
+Prompting Index (HPI), which demonstrates the cognitive competencies of LLMs
+across diverse datasets and offers insights into the cognitive demands that
+datasets place on different LLMs. This approach enables a comprehensive
+evaluation of an LLMs problem solving abilities and the intricacy of a dataset,
+offering a standardized metric for task complexity. Extensive experiments with
+multiple datasets and LLMs show that HPF enhances LLM performance by 2% to 63%
+compared to baseline performance, with GSM8k being the most cognitively complex
+task among reasoning and coding tasks with an average HPI of 3.20 confirming
+the effectiveness of HPT. To support future research and reproducibility in
+this domain, the implementations of HPT and HPF are available here.
+
+
+
+
+ + ♻ ☆ Probability of Differentiation Reveals Brittleness of Homogeneity Bias + in GPT-4 +
+ +
+ Homogeneity bias in Large Language Models (LLMs) refers to their tendency to
+homogenize the representations of some groups compared to others. Previous
+studies documenting this bias have predominantly used encoder models, which may
+have inadvertently introduced biases. To address this limitation, we prompted
+GPT-4 to generate single word/expression completions associated with 18
+situation cues-specific, measurable elements of environments that influence how
+individuals perceive situations and compared the variability of these
+completions using probability of differentiation. This approach directly
+assessed homogeneity bias from the model's outputs, bypassing encoder models.
+Across five studies, we find that homogeneity bias is highly volatile across
+situation cues and writing prompts, suggesting that the bias observed in past
+work may reflect those within encoder models rather than LLMs. Furthermore, we
+find that homogeneity bias in LLMs is brittle, as even minor and arbitrary
+changes in prompts can significantly alter the expression of biases. Future
+work should further explore how variations in syntactic features and topic
+choices in longer text generations influence homogeneity bias in LLMs.
+
+
+
+
+ + ♻ ☆ PediaBench: A Comprehensive Chinese Pediatric Dataset for Benchmarking + Large Language Models +
+
+
+
+
+
+
+
+ Qian Zhang, Panfeng Chen, Jiali Li, Linkun Feng, Shuyu Liu, Heng Zhao, Mei Chen, Hui Li, Yanhao Wang
+
+
+ The emergence of Large Language Models (LLMs) in the medical domain has
+stressed a compelling need for standard datasets to evaluate their
+question-answering (QA) performance. Although there have been several benchmark
+datasets for medical QA, they either cover common knowledge across different
+departments or are specific to another department rather than pediatrics.
+Moreover, some of them are limited to objective questions and do not measure
+the generation capacity of LLMs. Therefore, they cannot comprehensively assess
+the QA ability of LLMs in pediatrics. To fill this gap, we construct
+PediaBench, the first Chinese pediatric dataset for LLM evaluation.
+Specifically, it contains 4,565 objective questions and 1,632 subjective
+questions spanning 12 pediatric disease groups. It adopts an integrated scoring
+criterion based on different difficulty levels to thoroughly assess the
+proficiency of an LLM in instruction following, knowledge understanding,
+clinical case analysis, etc. Finally, we validate the effectiveness of
+PediaBench with extensive experiments on 20 open-source and commercial LLMs.
+Through an in-depth analysis of experimental results, we offer insights into
+the ability of LLMs to answer pediatric questions in the Chinese context,
+highlighting their limitations for further improvements. Our code and data are
+published at https://github.com/ACMISLab/PediaBench.
+
+
+
+ comment: 21 pages, 12 figures
+
+
+
+ + ♻ ☆ From Generation to Judgment: Opportunities and Challenges of + LLM-as-a-judge +
+
+
+
+
+
+
+
+ Dawei Li, Bohan Jiang, Liangjie Huang, Alimohammad Beigi, Chengshuai Zhao, Zhen Tan, Amrita Bhattacharjee, Yuxuan Jiang, Canyu Chen, Tianhao Wu, Kai Shu, Lu Cheng, Huan Liu
+
+
+ Assessment and evaluation have long been critical challenges in artificial
+intelligence (AI) and natural language processing (NLP). However, traditional
+methods, whether matching-based or embedding-based, often fall short of judging
+subtle attributes and delivering satisfactory results. Recent advancements in
+Large Language Models (LLMs) inspire the "LLM-as-a-judge" paradigm, where LLMs
+are leveraged to perform scoring, ranking, or selection across various tasks
+and applications. This paper provides a comprehensive survey of LLM-based
+judgment and assessment, offering an in-depth overview to advance this emerging
+field. We begin by giving detailed definitions from both input and output
+perspectives. Then we introduce a comprehensive taxonomy to explore
+LLM-as-a-judge from three dimensions: what to judge, how to judge and where to
+judge. Finally, we compile benchmarks for evaluating LLM-as-a-judge and
+highlight key challenges and promising directions, aiming to provide valuable
+insights and inspire future research in this promising research area. Paper
+list and more resources about LLM-as-a-judge can be found at
+\url{https://github.com/llm-as-a-judge/Awesome-LLM-as-a-judge} and
+\url{https://llm-as-a-judge.github.io}.
+
+
+
+ comment: v3: add missing citations; 32 pages, 5 figures
+
+