+
+ + Computation and Language 72 +
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+
+ + ☆ TeamCraft: A Benchmark for Multi-Modal Multi-Agent Systems in Minecraft +
+ +
+ Collaboration is a cornerstone of society. In the real world, human teammates
+make use of multi-sensory data to tackle challenging tasks in ever-changing
+environments. It is essential for embodied agents collaborating in
+visually-rich environments replete with dynamic interactions to understand
+multi-modal observations and task specifications. To evaluate the performance
+of generalizable multi-modal collaborative agents, we present TeamCraft, a
+multi-modal multi-agent benchmark built on top of the open-world video game
+Minecraft. The benchmark features 55,000 task variants specified by multi-modal
+prompts, procedurally-generated expert demonstrations for imitation learning,
+and carefully designed protocols to evaluate model generalization capabilities.
+We also perform extensive analyses to better understand the limitations and
+strengths of existing approaches. Our results indicate that existing models
+continue to face significant challenges in generalizing to novel goals, scenes,
+and unseen numbers of agents. These findings underscore the need for further
+research in this area. The TeamCraft platform and dataset are publicly
+available at https://github.com/teamcraft-bench/teamcraft.
+
+
+
+
+ + ☆ Uncertainty Quantification for Transformer Models for Dark-Pattern + Detection +
+
+
+
+
+
+
+
+ Javier Muñoz, Álvaro Huertas-García, Carlos Martí-González, Enrique De Miguel Ambite
+
+
+ The opaque nature of transformer-based models, particularly in applications
+susceptible to unethical practices such as dark-patterns in user interfaces,
+requires models that integrate uncertainty quantification to enhance trust in
+predictions. This study focuses on dark-pattern detection, deceptive design
+choices that manipulate user decisions, undermining autonomy and consent. We
+propose a differential fine-tuning approach implemented at the final
+classification head via uncertainty quantification with transformer-based
+pre-trained models. Employing a dense neural network (DNN) head architecture as
+a baseline, we examine two methods capable of quantifying uncertainty:
+Spectral-normalized Neural Gaussian Processes (SNGPs) and Bayesian Neural
+Networks (BNNs). These methods are evaluated on a set of open-source
+foundational models across multiple dimensions: model performance, variance in
+certainty of predictions and environmental impact during training and inference
+phases. Results demonstrate that integrating uncertainty quantification
+maintains performance while providing insights into challenging instances
+within the models. Moreover, the study reveals that the environmental impact
+does not uniformly increase with the incorporation of uncertainty
+quantification techniques. The study's findings demonstrate that uncertainty
+quantification enhances transparency and provides measurable confidence in
+predictions, improving the explainability and clarity of black-box models. This
+facilitates informed decision-making and mitigates the influence of
+dark-patterns on user interfaces. These results highlight the importance of
+incorporating uncertainty quantification techniques in developing machine
+learning models, particularly in domains where interpretability and
+trustworthiness are critical.
+
+
+
+
+ + ☆ Enhancing FKG.in: automating Indian food composition analysis +
+ +
+ This paper presents a novel approach to compute food composition data for
+Indian recipes using a knowledge graph for Indian food (FKG.in) and LLMs. The
+primary focus is to provide a broad overview of an automated food composition
+analysis workflow and describe its core functionalities: nutrition data
+aggregation, food composition analysis, and LLM-augmented information
+resolution. This workflow aims to complement FKG.in and iteratively supplement
+food composition data from verified knowledge bases. Additionally, this paper
+highlights the challenges of representing Indian food and accessing food
+composition data digitally. It also reviews three key sources of food
+composition data: the Indian Food Composition Tables, the Indian Nutrient
+Databank, and the Nutritionix API. Furthermore, it briefly outlines how users
+can interact with the workflow to obtain diet-based health recommendations and
+detailed food composition information for numerous recipes. We then explore the
+complex challenges of analyzing Indian recipe information across dimensions
+such as structure, multilingualism, and uncertainty as well as present our
+ongoing work on LLM-based solutions to address these issues. The methods
+proposed in this workshop paper for AI-driven knowledge curation and
+information resolution are application-agnostic, generalizable, and replicable
+for any domain.
+
+
+
+ comment: 15 pages, 3 figures, 30 references, International Conference on
+ Pattern Recognition 2024 - Multimedia Assisted Dietary Management Workshop
+
+
+
+ + ☆ MAmmoTH-VL: Eliciting Multimodal Reasoning with Instruction Tuning at + Scale +
+
+
+
+
+
+
+
+ Jarvis Guo, Tuney Zheng, Yuelin Bai, Bo Li, Yubo Wang, King Zhu, Yizhi Li, Graham Neubig, Wenhu Chen, Xiang Yue
+
+
+ Open-source multimodal large language models (MLLMs) have shown significant
+potential in a broad range of multimodal tasks. However, their reasoning
+capabilities remain constrained by existing instruction-tuning datasets, which
+were predominately repurposed from academic datasets such as VQA, AI2D, and
+ChartQA. These datasets target simplistic tasks, and only provide phrase-level
+answers without any intermediate rationales. To address these challenges, we
+introduce a scalable and cost-effective method to construct a large-scale
+multimodal instruction-tuning dataset with rich intermediate rationales
+designed to elicit CoT reasoning. Using only open models, we create a dataset
+containing 12M instruction-response pairs to cover diverse, reasoning-intensive
+tasks with detailed and faithful rationales. Experiments demonstrate that
+training MLLMs on this dataset significantly improves reasoning capabilities,
+achieving state-of-the-art performance on benchmarks such as MathVerse (+8.1%),
+MMMU-Pro (+7%), and MuirBench (+13.3%). Additionally, the model demonstrates
+notable improvements of up to 4% on non-reasoning-based benchmarks. Ablation
+studies further highlight the importance of key components, such as rewriting
+and self-filtering, in the dataset construction process.
+
+
+
+
+ + ☆ LIAR: Leveraging Alignment (Best-of-N) to Jailbreak LLMs in Seconds +
+
+
+
+
+
+
+
+ James Beetham, Souradip Chakraborty, Mengdi Wang, Furong Huang, Amrit Singh Bedi, Mubarak Shah
+
+
+ Many existing jailbreak techniques rely on solving discrete combinatorial
+optimization, while more recent approaches involve training LLMs to generate
+multiple adversarial prompts. However, both approaches require significant
+computational resources to produce even a single adversarial prompt. We
+hypothesize that the inefficiency of current approaches stems from an
+inadequate characterization of the jailbreak problem. To address this gap, we
+formulate the jailbreak problem in terms of alignment. By starting from an
+available safety-aligned model, we leverage an unsafe reward to guide the safe
+model towards generating unsafe outputs using alignment techniques (e.g.,
+reinforcement learning from human feedback), effectively performing
+jailbreaking via alignment. We propose a novel jailbreak method called LIAR
+(LeveragIng Alignment to jailbReak). To demonstrate the simplicity and
+effectiveness of our approach, we employ a best-of-N method to solve the
+alignment problem. LIAR offers significant advantages: lower computational
+requirements without additional training, fully black-box operation,
+competitive attack success rates, and more human-readable prompts. We provide
+theoretical insights into the possibility of jailbreaking a safety-aligned
+model, revealing inherent vulnerabilities in current alignment strategies for
+LLMs. We also provide sub-optimality guarantees for the proposed \algo.
+Experimentally, we achieve ASR comparable to the SoTA with a 10x improvement to
+perplexity and a Time-to-Attack measured in seconds rather than tens of hours.
+
+
+
+
+ + ☆ BEExformer: A Fast Inferencing Transformer Architecture via Binarization + with Multiple Early Exits +
+ +
+ Large Language Models (LLMs) based on transformers achieve cutting-edge
+results on a variety of applications. However, their enormous size and
+processing requirements make deployment on devices with constrained resources
+extremely difficult. Among various efficiency considerations, model
+binarization and Early Exit (EE) are common effective solutions. However,
+binarization may lead to performance loss due to reduced precision affecting
+gradient estimation and parameter updates. Besides, the present early-exit
+mechanisms are still in the nascent stages of research. To ameliorate these
+issues, we propose Binarized Early Exit Transformer (BEExformer), the
+first-ever selective learning transformer architecture to combine early exit
+with binarization for textual inference. It improves the binarization process
+through a differentiable second-order approximation to the impulse function.
+This enables gradient computation concerning both the sign as well as the
+magnitude of the weights. In contrast to absolute threshold-based EE, the
+proposed EE mechanism hinges on fractional reduction in entropy among
+intermediate transformer blocks with soft-routing loss estimation. While
+binarization results in 18.44 times reduction in model size, early exit reduces
+the FLOPs during inference by 54.85% and even improves accuracy by 5.98%
+through resolving the "overthinking" problem inherent in deep networks.
+Moreover, the proposed BEExformer simplifies training by not requiring
+knowledge distillation from a full-precision LLM. Extensive evaluation on the
+GLUE dataset and comparison with the SOTA works showcase its pareto-optimal
+performance-efficiency trade-off.
+
+
+
+ comment: 15 pages, 15 figures, 3 tables
+
+
+
+ + ☆ 100% Hallucination Elimination Using Acurai +
+ +
+ The issue of hallucinations in large language models (LLMs) remains a
+critical barrier to the adoption of AI in enterprise and other high-stakes
+applications. Despite advancements in retrieval-augmented generation (RAG)
+systems, current state-of-the-art methods fail to achieve more than 80%
+accuracy in generating faithful and factually correct outputs, even when
+provided with relevant and accurate context. In this work, we introduce Acurai,
+a novel systematic approach that achieves 100% hallucination-free responses in
+LLMs by reformatting queries and context data prior to input. Leveraging a deep
+understanding of LLM internal representations, the importance of noun-phrase
+dominance, and the role of discrete functional units (DFUs), Acurai ensures
+alignment between input context and generated output. We validate this method
+using the RAGTruth corpus, demonstrating its ability to eliminate 100%
+hallucinations for both GPT-4 and GPT-3.5 Turbo. Acurai sets a new standard for
+achieving consistent, accurate, and faithful AI responses, marking a
+significant step forward in the development of trustworthy AI systems.
+
+
+
+
+ + ☆ Evaluating and Aligning CodeLLMs on Human Preference +
+
+
+
+
+
+
+
+ Jian Yang, Jiaxi Yang, Ke Jin, Yibo Miao, Lei Zhang, Liqun Yang, Zeyu Cui, Yichang Zhang, Binyuan Hui, Junyang Lin
+
+
+ Code large language models (codeLLMs) have made significant strides in code
+generation. Most previous code-related benchmarks, which consist of various
+programming exercises along with the corresponding test cases, are used as a
+common measure to evaluate the performance and capabilities of code LLMs.
+However, the current code LLMs focus on synthesizing the correct code snippet,
+ignoring the alignment with human preferences, where the query should be
+sampled from the practical application scenarios and the model-generated
+responses should satisfy the human preference. To bridge the gap between the
+model-generated response and human preference, we present a rigorous
+human-curated benchmark CodeArena to emulate the complexity and diversity of
+real-world coding tasks, where 397 high-quality samples spanning 40 categories
+and 44 programming languages, carefully curated from user queries. Further, we
+propose a diverse synthetic instruction corpus SynCode-Instruct (nearly 20B
+tokens) by scaling instructions from the website to verify the effectiveness of
+the large-scale synthetic instruction fine-tuning, where Qwen2.5-SynCoder
+totally trained on synthetic instruction data can achieve top-tier performance
+of open-source code LLMs. The results find performance differences between
+execution-based benchmarks and CodeArena. Our systematic experiments of
+CodeArena on 40+ LLMs reveal a notable performance gap between open SOTA code
+LLMs (e.g. Qwen2.5-Coder) and proprietary LLMs (e.g., OpenAI o1), underscoring
+the importance of the human preference
+alignment.\footnote{\url{https://codearenaeval.github.io/ }}
+
+
+
+
+ + ☆ ConQRet: Benchmarking Fine-Grained Evaluation of Retrieval Augmented + Argumentation with LLM Judges +
+ +
+ Computational argumentation, which involves generating answers or summaries
+for controversial topics like abortion bans and vaccination, has become
+increasingly important in today's polarized environment. Sophisticated LLM
+capabilities offer the potential to provide nuanced, evidence-based answers to
+such questions through Retrieval-Augmented Argumentation (RAArg), leveraging
+real-world evidence for high-quality, grounded arguments. However, evaluating
+RAArg remains challenging, as human evaluation is costly and difficult for
+complex, lengthy answers on complicated topics. At the same time, re-using
+existing argumentation datasets is no longer sufficient, as they lack long,
+complex arguments and realistic evidence from potentially misleading sources,
+limiting holistic evaluation of retrieval effectiveness and argument quality.
+To address these gaps, we investigate automated evaluation methods using
+multiple fine-grained LLM judges, providing better and more interpretable
+assessments than traditional single-score metrics and even previously reported
+human crowdsourcing. To validate the proposed techniques, we introduce ConQRet,
+a new benchmark featuring long and complex human-authored arguments on debated
+topics, grounded in real-world websites, allowing an exhaustive evaluation
+across retrieval effectiveness, argument quality, and groundedness. We validate
+our LLM Judges on a prior dataset and the new ConQRet benchmark. Our proposed
+LLM Judges and the ConQRet benchmark can enable rapid progress in computational
+argumentation and can be naturally extended to other complex
+retrieval-augmented generation tasks.
+
+
+
+
+ + ☆ QueEn: A Large Language Model for Quechua-English Translation +
+
+
+
+
+
+
+
+ Junhao Chen, Peng Shu, Yiwei Li, Huaqin Zhao, Hanqi Jiang, Yi Pan, Yifan Zhou, Zhengliang Liu, Lewis C Howe, Tianming Liu
+
+
+ Recent studies show that large language models (LLMs) are powerful tools for
+working with natural language, bringing advances in many areas of computational
+linguistics. However, these models face challenges when applied to low-resource
+languages due to limited training data and difficulty in understanding cultural
+nuances. In this paper, we propose QueEn, a novel approach for Quechua-English
+translation that combines Retrieval-Augmented Generation (RAG) with
+parameter-efficient fine-tuning techniques. Our method leverages external
+linguistic resources through RAG and uses Low-Rank Adaptation (LoRA) for
+efficient model adaptation. Experimental results show that our approach
+substantially exceeds baseline models, with a BLEU score of 17.6 compared to
+1.5 for standard GPT models. The integration of RAG with fine-tuning allows our
+system to address the challenges of low-resource language translation while
+maintaining computational efficiency. This work contributes to the broader goal
+of preserving endangered languages through advanced language technologies.
+
+
+
+
+ + ☆ Benchmarking Open-ended Audio Dialogue Understanding for Large + Audio-Language Models +
+ +
+ Large Audio-Language Models (LALMs) have unclocked audio dialogue
+capabilities, where audio dialogues are a direct exchange of spoken language
+between LALMs and humans. Recent advances, such as GPT-4o, have enabled LALMs
+in back-and-forth audio dialogues with humans. This progression not only
+underscores the potential of LALMs but also broadens their applicability across
+a wide range of practical scenarios supported by audio dialogues. However,
+given these advancements, a comprehensive benchmark to evaluate the performance
+of LALMs in the open-ended audio dialogue understanding remains absent
+currently. To address this gap, we propose an Audio Dialogue Understanding
+Benchmark (ADU-Bench), which consists of 4 benchmark datasets. They assess the
+open-ended audio dialogue ability for LALMs in 3 general scenarios, 12 skills,
+9 multilingual languages, and 4 categories of ambiguity handling. Notably, we
+firstly propose the evaluation of ambiguity handling in audio dialogues that
+expresses different intentions beyond the same literal meaning of sentences,
+e.g., "Really!?" with different intonations. In summary, ADU-Bench includes
+over 20,000 open-ended audio dialogues for the assessment of LALMs. Through
+extensive experiments conducted on 13 LALMs, our analysis reveals that there is
+still considerable room for improvement in the audio dialogue understanding
+abilities of existing LALMs. In particular, they struggle with mathematical
+symbols and formulas, understanding human behavior such as roleplay,
+comprehending multiple languages, and handling audio dialogue ambiguities from
+different phonetic elements, such as intonations, pause positions, and
+homophones.
+
+
+
+
+ + ☆ Multimodal Fact-Checking with Vision Language Models: A Probing + Classifier based Solution with Embedding Strategies COLING2025 +
+ +
+ This study evaluates the effectiveness of Vision Language Models (VLMs) in
+representing and utilizing multimodal content for fact-checking. To be more
+specific, we investigate whether incorporating multimodal content improves
+performance compared to text-only models and how well VLMs utilize text and
+image information to enhance misinformation detection. Furthermore we propose a
+probing classifier based solution using VLMs. Our approach extracts embeddings
+from the last hidden layer of selected VLMs and inputs them into a neural
+probing classifier for multi-class veracity classification. Through a series of
+experiments on two fact-checking datasets, we demonstrate that while
+multimodality can enhance performance, fusing separate embeddings from text and
+image encoders yielded superior results compared to using VLM embeddings.
+Furthermore, the proposed neural classifier significantly outperformed KNN and
+SVM baselines in leveraging extracted embeddings, highlighting its
+effectiveness for multimodal fact-checking.
+
+
+
+ comment: Accepted to COLING2025
+
+
+
+ + ☆ Findings of the Second BabyLM Challenge: Sample-Efficient Pretraining on + Developmentally Plausible Corpora +
+
+
+
+
+
+
+
+ Michael Y. Hu, Aaron Mueller, Candace Ross, Adina Williams, Tal Linzen, Chengxu Zhuang, Ryan Cotterell, Leshem Choshen, Alex Warstadt, Ethan Gotlieb Wilcox
+
+
+ The BabyLM Challenge is a community effort to close the data-efficiency gap
+between human and computational language learners. Participants compete to
+optimize language model training on a fixed language data budget of 100 million
+words or less. This year, we released improved text corpora, as well as a
+vision-and-language corpus to facilitate research into cognitively plausible
+vision language models. Submissions were compared on evaluation tasks targeting
+grammatical ability, (visual) question answering, pragmatic abilities, and
+grounding, among other abilities. Participants could submit to a 10M-word
+text-only track, a 100M-word text-only track, and/or a 100M-word and image
+multimodal track. From 31 submissions employing diverse methods, a hybrid
+causal-masked language model architecture outperformed other approaches. No
+submissions outperformed the baselines in the multimodal track. In follow-up
+analyses, we found a strong relationship between training FLOPs and average
+performance across tasks, and that the best-performing submissions proposed
+changes to the training data, training objective, and model architecture. This
+year's BabyLM Challenge shows that there is still significant room for
+innovation in this setting, in particular for image-text modeling, but
+community-driven research can yield actionable insights about effective
+strategies for small-scale language modeling.
+
+
+
+
+ + ☆ Explingo: Explaining AI Predictions using Large Language Models +
+
+
+
+
+
+
+
+ Alexandra Zytek, Sara Pido, Sarah Alnegheimish, Laure Berti-Equille, Kalyan Veeramachaneni
+
+
+ Explanations of machine learning (ML) model predictions generated by
+Explainable AI (XAI) techniques such as SHAP are essential for people using ML
+outputs for decision-making. We explore the potential of Large Language Models
+(LLMs) to transform these explanations into human-readable, narrative formats
+that align with natural communication. We address two key research questions:
+(1) Can LLMs reliably transform traditional explanations into high-quality
+narratives? and (2) How can we effectively evaluate the quality of narrative
+explanations? To answer these questions, we introduce Explingo, which consists
+of two LLM-based subsystems, a Narrator and Grader. The Narrator takes in ML
+explanations and transforms them into natural-language descriptions. The Grader
+scores these narratives on a set of metrics including accuracy, completeness,
+fluency, and conciseness.
+ Our experiments demonstrate that LLMs can generate high-quality narratives
+that achieve high scores across all metrics, particularly when guided by a
+small number of human-labeled and bootstrapped examples. We also identified
+areas that remain challenging, in particular for effectively scoring narratives
+in complex domains. The findings from this work have been integrated into an
+open-source tool that makes narrative explanations available for further
+applications.
+
+
+
+ comment: To be presented in the 2024 IEEE International Conference on Big Data
+ (IEEE BigData)
+
+
+
+ + ☆ A Practical Examination of AI-Generated Text Detectors for Large + Language Models +
+ +
+ The proliferation of large language models has raised growing concerns about
+their misuse, particularly in cases where AI-generated text is falsely
+attributed to human authors. Machine-generated content detectors claim to
+effectively identify such text under various conditions and from any language
+model. This paper critically evaluates these claims by assessing several
+popular detectors (RADAR, Wild, T5Sentinel, Fast-DetectGPT, GPTID, LogRank,
+Binoculars) on a range of domains, datasets, and models that these detectors
+have not previously encountered. We employ various prompting strategies to
+simulate adversarial attacks, demonstrating that even moderate efforts can
+significantly evade detection. We emphasize the importance of the true positive
+rate at a specific false positive rate (TPR@FPR) metric and demonstrate that
+these detectors perform poorly in certain settings, with TPR@.01 as low as 0\%.
+Our findings suggest that both trained and zero-shot detectors struggle to
+maintain high sensitivity while achieving a reasonable true positive rate.
+
+
+
+ comment: 8 pages. Submitted to ARR October cycle
+
+
+
+ + ☆ Unifying Dual-Space Embedding for Entity Alignment via Contrastive + Learning COLING2025 +
+ +
+ Entity alignment aims to match identical entities across different knowledge
+graphs (KGs). Graph neural network-based entity alignment methods have achieved
+promising results in Euclidean space. However, KGs often contain complex
+structures, including both local and hierarchical ones, which make it
+challenging to efficiently represent them within a single space. In this paper,
+we proposed a novel method UniEA, which unifies dual-space embedding to
+preserve the intrinsic structure of KGs. Specifically, we learn graph structure
+embedding in both Euclidean and hyperbolic spaces simultaneously to maximize
+the consistency between the embedding in both spaces. Moreover, we employ
+contrastive learning to mitigate the misalignment issues caused by similar
+entities, where embedding of similar neighboring entities within the KG become
+too close in distance. Extensive experiments on benchmark datasets demonstrate
+that our method achieves state-of-the-art performance in structure-based EA.
+Our code is available at https://github.com/wonderCS1213/UniEA.
+
+
+
+ comment: Accepted by COLING2025
+
+
+
+ + ☆ Steps are all you need: Rethinking STEM Education with Prompt + Engineering +
+
+
+
+
+
+
+
+ Krishnasai Addala, Kabir Dev Paul Baghel, Chhavi Kirtani, Avinash Anand, Rajiv Ratn Shah
+
+
+ Few shot and Chain-of-Thought prompting have shown promise when applied to
+Physics Question Answering Tasks, but are limited by the lack of mathematical
+ability inherent to LLMs, and are prone to hallucination. By utilizing a
+Mixture of Experts (MoE) Model, along with analogical prompting, we are able to
+show improved model performance when compared to the baseline on standard LLMs.
+We also survey the limits of these prompting techniques and the effects they
+have on model performance. Additionally, we propose Analogical CoT prompting, a
+prompting technique designed to allow smaller, open source models to leverage
+Analogical prompting, something they have struggled with, possibly due to a
+lack of specialist training data.
+
+
+
+
+ + ☆ PETapter: Leveraging PET-style classification heads for modular few-shot + parameter-efficient fine-tuning +
+ +
+ Few-shot learning and parameter-efficient fine-tuning (PEFT) are crucial to
+overcome the challenges of data scarcity and ever growing language model sizes.
+This applies in particular to specialized scientific domains, where researchers
+might lack expertise and resources to fine-tune high-performing language models
+to nuanced tasks. We propose PETapter, a novel method that effectively combines
+PEFT methods with PET-style classification heads to boost few-shot learning
+capabilities without the significant computational overhead typically
+associated with full model training. We validate our approach on three
+established NLP benchmark datasets and one real-world dataset from
+communication research. We show that PETapter not only achieves comparable
+performance to full few-shot fine-tuning using pattern-exploiting training
+(PET), but also provides greater reliability and higher parameter efficiency
+while enabling higher modularity and easy sharing of the trained modules, which
+enables more researchers to utilize high-performing NLP-methods in their
+research.
+
+
+
+
+ + ☆ Gla-AI4BioMed at RRG24: Visual Instruction-tuned Adaptation for + Radiology Report Generation ACL 2024 +
+ +
+ We introduce a radiology-focused visual language model designed to generate
+radiology reports from chest X-rays. Building on previous findings that large
+language models (LLMs) can acquire multimodal capabilities when aligned with
+pretrained vision encoders, we demonstrate similar potential with chest X-ray
+images. This integration enhances the ability of model to understand and
+describe chest X-ray images. Our model combines an image encoder with a
+fine-tuned LLM based on the Vicuna-7B architecture, enabling it to generate
+different sections of a radiology report with notable accuracy. The training
+process involves a two-stage approach: (i) initial alignment of chest X-ray
+features with the LLM (ii) followed by fine-tuning for radiology report
+generation.
+
+
+
+ comment: Accepted by BioNLP@ACL 2024
+
+
+
+ + ☆ KaLM: Knowledge-aligned Autoregressive Language Modeling via Dual-view + Knowledge Graph Contrastive Learning +
+ +
+ Autoregressive large language models (LLMs) pre-trained by next token
+prediction are inherently proficient in generative tasks. However, their
+performance on knowledge-driven tasks such as factual knowledge querying
+remains unsatisfactory. Knowledge graphs (KGs), as high-quality structured
+knowledge bases, can provide reliable knowledge for LLMs, potentially
+compensating for their knowledge deficiencies. Aligning LLMs with explicit,
+structured knowledge from KGs has been a challenge; previous attempts either
+failed to effectively align knowledge representations or compromised the
+generative capabilities of LLMs, leading to less-than-optimal outcomes. This
+paper proposes \textbf{KaLM}, a \textit{Knowledge-aligned Language Modeling}
+approach, which fine-tunes autoregressive LLMs to align with KG knowledge via
+the joint objective of explicit knowledge alignment and implicit knowledge
+alignment. The explicit knowledge alignment objective aims to directly optimize
+the knowledge representation of LLMs through dual-view knowledge graph
+contrastive learning. The implicit knowledge alignment objective focuses on
+incorporating textual patterns of knowledge into LLMs through triple completion
+language modeling. Notably, our method achieves a significant performance boost
+in evaluations of knowledge-driven tasks, specifically embedding-based
+knowledge graph completion and generation-based knowledge graph question
+answering.
+
+
+
+
+ + ☆ C$^2$LEVA: Toward Comprehensive and Contamination-Free Language Model + Evaluation +
+
+
+
+
+
+
+
+ Yanyang Li, Tin Long Wong, Cheung To Hung, Jianqiao Zhao, Duo Zheng, Ka Wai Liu, Michael R. Lyu, Liwei Wang
+
+
+ Recent advances in large language models (LLMs) have shown significant
+promise, yet their evaluation raises concerns, particularly regarding data
+contamination due to the lack of access to proprietary training data. To
+address this issue, we present C$^2$LEVA, a comprehensive bilingual benchmark
+featuring systematic contamination prevention. C$^2$LEVA firstly offers a
+holistic evaluation encompassing 22 tasks, each targeting a specific
+application or ability of LLMs, and secondly a trustworthy assessment due to
+our contamination-free tasks, ensured by a systematic contamination prevention
+strategy that fully automates test data renewal and enforces data protection
+during benchmark data release. Our large-scale evaluation of 15 open-source and
+proprietary models demonstrates the effectiveness of C$^2$LEVA.
+
+
+
+
+ + ☆ A Federated Approach to Few-Shot Hate Speech Detection for Marginalized + Communities +
+ +
+ Hate speech online remains an understudied issue for marginalized
+communities, and has seen rising relevance, especially in the Global South,
+which includes developing societies with increasing internet penetration. In
+this paper, we aim to provide marginalized communities living in societies
+where the dominant language is low-resource with a privacy-preserving tool to
+protect themselves from hate speech on the internet by filtering offensive
+content in their native languages. Our contribution in this paper is twofold:
+1) we release REACT (REsponsive hate speech datasets Across ConTexts), a
+collection of high-quality, culture-specific hate speech detection datasets
+comprising seven distinct target groups in eight low-resource languages,
+curated by experienced data collectors; 2) we propose a solution to few-shot
+hate speech detection utilizing federated learning (FL), a privacy-preserving
+and collaborative learning approach, to continuously improve a central model
+that exhibits robustness when tackling different target groups and languages.
+By keeping the training local to the users' devices, we ensure the privacy of
+the users' data while benefitting from the efficiency of federated learning.
+Furthermore, we personalize client models to target-specific training data and
+evaluate their performance. Our results indicate the effectiveness of FL across
+different target groups, whereas the benefits of personalization on few-shot
+learning are not clear.
+
+
+
+
+ + ☆ Who Speaks Next? Multi-party AI Discussion Leveraging the Systematics of + Turn-taking in Murder Mystery Games +
+ +
+ Multi-agent systems utilizing large language models (LLMs) have shown great
+promise in achieving natural dialogue. However, smooth dialogue control and
+autonomous decision making among agents still remain challenges. In this study,
+we focus on conversational norms such as adjacency pairs and turn-taking found
+in conversation analysis and propose a new framework called "Murder Mystery
+Agents" that applies these norms to AI agents' dialogue control. As an
+evaluation target, we employed the "Murder Mystery" game, a reasoning-type
+table-top role-playing game that requires complex social reasoning and
+information manipulation. In this game, players need to unravel the truth of
+the case based on fragmentary information through cooperation and bargaining.
+The proposed framework integrates next speaker selection based on adjacency
+pairs and a self-selection mechanism that takes agents' internal states into
+account to achieve more natural and strategic dialogue. To verify the
+effectiveness of this new approach, we analyzed utterances that led to dialogue
+breakdowns and conducted automatic evaluation using LLMs, as well as human
+evaluation using evaluation criteria developed for the Murder Mystery game.
+Experimental results showed that the implementation of the next speaker
+selection mechanism significantly reduced dialogue breakdowns and improved the
+ability of agents to share information and perform logical reasoning. The
+results of this study demonstrate that the systematics of turn-taking in human
+conversation are also effective in controlling dialogue among AI agents, and
+provide design guidelines for more advanced multi-agent dialogue systems.
+
+
+
+
+ + ☆ Probing the contents of semantic representations from text, behavior, + and brain data using the psychNorms metabase +
+ +
+ Semantic representations are integral to natural language processing,
+psycholinguistics, and artificial intelligence. Although often derived from
+internet text, recent years have seen a rise in the popularity of
+behavior-based (e.g., free associations) and brain-based (e.g., fMRI)
+representations, which promise improvements in our ability to measure and model
+human representations. We carry out the first systematic evaluation of the
+similarities and differences between semantic representations derived from
+text, behavior, and brain data. Using representational similarity analysis, we
+show that word vectors derived from behavior and brain data encode information
+that differs from their text-derived cousins. Furthermore, drawing on our
+psychNorms metabase, alongside an interpretability method that we call
+representational content analysis, we find that, in particular, behavior
+representations capture unique variance on certain affective, agentic, and
+socio-moral dimensions. We thus establish behavior as an important complement
+to text for capturing human representations and behavior. These results are
+broadly relevant to research aimed at learning human-aligned semantic
+representations, including work on evaluating and aligning large language
+models.
+
+
+
+ comment: 13 pages, 5 figures, 2 tables
+
+
+
+ + ☆ Large Language Models for Ingredient Substitution in Food Recipes using + Supervised Fine-tuning and Direct Preference Optimization +
+
+
+
+
+
+
+
+ Thevin Senath, Kumuthu Athukorala, Ransika Costa, Surangika Ranathunga, Rishemjit Kaur
+
+
+ In this paper, we address the challenge of recipe personalization through
+ingredient substitution. We make use of Large Language Models (LLMs) to build
+an ingredient substitution system designed to predict plausible substitute
+ingredients within a given recipe context. Given that the use of LLMs for this
+task has been barely done, we carry out an extensive set of experiments to
+determine the best LLM, prompt, and the fine-tuning setups. We further
+experiment with methods such as multi-task learning, two-stage fine-tuning, and
+Direct Preference Optimization (DPO). The experiments are conducted using the
+publicly available Recipe1MSub corpus. The best results are produced by the
+Mistral7-Base LLM after fine-tuning and DPO. This result outperforms the strong
+baseline available for the same corpus with a Hit@1 score of 22.04. Thus we
+believe that this research represents a significant step towards enabling
+personalized and creative culinary experiences by utilizing LLM-based
+ingredient substitution.
+
+
+
+
+ + ☆ DEMO: Reframing Dialogue Interaction with Fine-grained Element Modeling +
+
+
+
+
+
+
+
+ Minzheng Wang, Xinghua Zhang, Kun Chen, Nan Xu, Haiyang Yu, Fei Huang, Wenji Mao, Yongbin Li
+
+
+ Large language models (LLMs) have made dialogue one of the central modes of
+human-machine interaction, leading to the accumulation of vast amounts of
+conversation logs and increasing demand for dialogue generation. A
+conversational life-cycle spans from the Prelude through the Interlocution to
+the Epilogue, encompassing various elements. Despite the existence of numerous
+dialogue-related studies, there is a lack of benchmarks that encompass
+comprehensive dialogue elements, hindering precise modeling and systematic
+evaluation. To bridge this gap, we introduce an innovative research task
+$\textbf{D}$ialogue $\textbf{E}$lement $\textbf{MO}$deling, including
+$\textit{Element Awareness}$ and $\textit{Dialogue Agent Interaction}$, and
+propose a novel benchmark, $\textbf{DEMO}$, designed for a comprehensive
+dialogue modeling and assessment. Inspired by imitation learning, we further
+build the agent which possesses the adept ability to model dialogue elements
+based on the DEMO benchmark. Extensive experiments indicate that existing LLMs
+still exhibit considerable potential for enhancement, and our DEMO agent has
+superior performance in both in-domain and out-of-domain tasks.
+
+
+
+ comment: We release the code and data at https://github.com/MozerWang/DEMO
+
+
+
+ + ☆ EACO: Enhancing Alignment in Multimodal LLMs via Critical Observation +
+
+
+
+
+
+
+
+ Yongxin Wang, Meng Cao, Haokun Lin, Mingfei Han, Liang Ma, Jin Jiang, Yuhao Cheng, Xiaodan Liang
+
+
+ Multimodal large language models (MLLMs) have achieved remarkable progress on
+various visual question answering and reasoning tasks leveraging instruction
+fine-tuning specific datasets. They can also learn from preference data
+annotated by human to enhance their reasoning ability and mitigate
+hallucinations. Most of preference data is generated from the model itself.
+However, existing methods require high-quality critical labels, which are
+costly and rely on human or proprietary models like GPT-4V. In this work, we
+propose Enhancing Alignment in MLLMs via Critical Observation (EACO), which
+aligns MLLMs by self-generated preference data using only 5k images
+economically. Our approach begins with collecting and refining a Scoring
+Evaluation Instruction-tuning dataset to train a critical evaluation model,
+termed the Critic. This Critic observes model responses across multiple
+dimensions, selecting preferred and non-preferred outputs for refined Direct
+Preference Optimization (DPO) tuning. To further enhance model performance, we
+employ an additional supervised fine-tuning stage after preference tuning. EACO
+reduces the overall hallucinations by 65.6% on HallusionBench and improves the
+reasoning ability by 21.8% on MME-Cognition. EACO achieves an 8.5% improvement
+over LLaVA-v1.6-Mistral-7B across multiple benchmarks. Remarkably, EACO also
+shows the potential critical ability in open-source MLLMs, demonstrating that
+EACO is a viable path to boost the competence of MLLMs.
+
+
+
+ comment: 19 pages
+
+
+
+ + ☆ Building a Family of Data Augmentation Models for Low-cost LLM + Fine-tuning on the Cloud +
+ +
+ Specializing LLMs in various domain-specific tasks has emerged as a critical
+step towards achieving high performance. However, the construction and
+annotation of datasets in specific domains are always very costly. Apart from
+using superior and expensive closed-source LLM APIs to construct datasets, some
+open-source models have become strong enough to handle dataset construction in
+many scenarios. Thus, we present a family of data augmentation models designed
+to significantly improve the efficiency for model fine-tuning. These models,
+trained based on sufficiently small LLMs, support key functionalities with low
+inference costs: instruction expansion, instruction refinement, and
+instruction-response pair expansion. To fulfill this goal, we first construct
+an automatic data collection system with seed datasets generated from both
+public repositories and our in-house datasets. This system leverages powerful
+LLMs to expand, refine and re-write the instructions and responses,
+incorporating quality assessment techniques. Following this, we introduce the
+training process of our models, which effectively distills task-solving and
+text synthesis abilities from teacher LLMs. Finally, we demonstrate how we
+integrate these functionalities into a machine learning platform to support
+low-cost LLM fine-tuning from both dataset preparation and training
+perspectives for users. Experiments and an application study prove the
+effectiveness of our approach.
+
+
+
+ comment: coling 2025 industry track
+
+
+
+ + ☆ EXAONE 3.5: Series of Large Language Models for Real-world Use Cases +
+
+
+
+
+
+
+
+ LG AI Research, Soyoung An, Kyunghoon Bae, Eunbi Choi, Kibong Choi, Stanley Jungkyu Choi, Seokhee Hong, Junwon Hwang, Hyojin Jeon, Gerrard Jeongwon Jo, Hyunjik Jo, Jiyeon Jung, Yountae Jung, Hyosang Kim, Joonkee Kim, Seonghwan Kim, Soyeon Kim, Sunkyoung Kim, Yireun Kim, Yongil Kim, Youchul Kim, Edward Hwayoung Lee, Haeju Lee, Honglak Lee, Jinsik Lee, Kyungmin Lee, Woohyung Lim, Sangha Park, Sooyoun Park, Yongmin Park, Sihoon Yang, Heuiyeen Yeen, Hyeongu Yun
+
+
+ This technical report introduces the EXAONE 3.5 instruction-tuned language
+models, developed and released by LG AI Research. The EXAONE 3.5 language
+models are offered in three configurations: 32B, 7.8B, and 2.4B. These models
+feature several standout capabilities: 1) exceptional instruction following
+capabilities in real-world scenarios, achieving the highest scores across seven
+benchmarks, 2) outstanding long-context comprehension, attaining the top
+performance in four benchmarks, and 3) competitive results compared to
+state-of-the-art open models of similar sizes across nine general benchmarks.
+The EXAONE 3.5 language models are open to anyone for research purposes and can
+be downloaded from https://huggingface.co/LGAI-EXAONE. For commercial use,
+please reach out to the official contact point of LG AI Research:
+contact_us@lgresearch.ai.
+
+
+
+ comment: arXiv admin note: text overlap with arXiv:2408.03541
+
+
+
+ + ☆ Breaking Event Rumor Detection via Stance-Separated Multi-Agent Debate +
+ +
+ The rapid spread of rumors on social media platforms during breaking events
+severely hinders the dissemination of the truth. Previous studies reveal that
+the lack of annotated resources hinders the direct detection of unforeseen
+breaking events not covered in yesterday's news. Leveraging large language
+models (LLMs) for rumor detection holds significant promise. However, it is
+challenging for LLMs to provide comprehensive responses to complex or
+controversial issues due to limited diversity. In this work, we propose the
+Stance Separated Multi-Agent Debate (S2MAD) to address this issue.
+Specifically, we firstly introduce Stance Separation, categorizing comments as
+either supporting or opposing the original claim. Subsequently, claims are
+classified as subjective or objective, enabling agents to generate reasonable
+initial viewpoints with different prompt strategies for each type of claim.
+Debaters then follow specific instructions through multiple rounds of debate to
+reach a consensus. If a consensus is not reached, a judge agent evaluates the
+opinions and delivers a final verdict on the claim's veracity. Extensive
+experiments conducted on two real-world datasets demonstrate that our proposed
+model outperforms state-of-the-art methods in terms of performance and
+effectively improves the performance of LLMs in breaking event rumor detection.
+
+
+
+
+ + ☆ Adaptive Dropout for Pruning Conformers +
+ +
+ This paper proposes a method to effectively perform joint
+training-and-pruning based on adaptive dropout layers with unit-wise retention
+probabilities. The proposed method is based on the estimation of a unit-wise
+retention probability in a dropout layer. A unit that is estimated to have a
+small retention probability can be considered to be prunable. The retention
+probability of the unit is estimated using back-propagation and the
+Gumbel-Softmax technique. This pruning method is applied at several application
+points in Conformers such that the effective number of parameters can be
+significantly reduced. Specifically, adaptive dropout layers are introduced in
+three locations in each Conformer block: (a) the hidden layer of the
+feed-forward-net component, (b) the query vectors and the value vectors of the
+self-attention component, and (c) the input vectors of the LConv component. The
+proposed method is evaluated by conducting a speech recognition experiment on
+the LibriSpeech task. It was shown that this approach could simultaneously
+achieve a parameter reduction and accuracy improvement. The word error rates
+improved by approx 1% while reducing the number of parameters by 54%.
+
+
+
+
+ + ☆ Rethinking Time Series Forecasting with LLMs via Nearest Neighbor + Contrastive Learning +
+ +
+ Adapting Large Language Models (LLMs) that are extensively trained on
+abundant text data, and customizing the input prompt to enable time series
+forecasting has received considerable attention. While recent work has shown
+great potential for adapting the learned prior of LLMs, the formulation of the
+prompt to finetune LLMs remains challenging as prompt should be aligned with
+time series data. Additionally, current approaches do not effectively leverage
+word token embeddings which embody the rich representation space learned by
+LLMs. This emphasizes the need for a robust approach to formulate the prompt
+which utilizes the word token embeddings while effectively representing the
+characteristics of the time series. To address these challenges, we propose
+NNCL-TLLM: Nearest Neighbor Contrastive Learning for Time series forecasting
+via LLMs. First, we generate time series compatible text prototypes such that
+each text prototype represents both word token embeddings in its neighborhood
+and time series characteristics via end-to-end finetuning. Next, we draw
+inspiration from Nearest Neighbor Contrastive Learning to formulate the prompt
+while obtaining the top-$k$ nearest neighbor time series compatible text
+prototypes. We then fine-tune the layer normalization and positional embeddings
+of the LLM, keeping the other layers intact, reducing the trainable parameters
+and decreasing the computational cost. Our comprehensive experiments
+demonstrate that NNCL-TLLM outperforms in few-shot forecasting while achieving
+competitive or superior performance over the state-of-the-art methods in
+long-term and short-term forecasting tasks.
+
+
+
+
+ + ☆ Direct Quantized Training of Language Models with Stochastic Rounding +
+
+
+
+
+
+
+
+ Kaiyan Zhao, Tsuguchika Tabaru, Kenichi Kobayashi, Takumi Honda, Masafumi Yamazaki, Yoshimasa Tsuruoka
+
+
+ Although recent quantized Large Language Models (LLMs), such as BitNet, have
+paved the way for significant reduction in memory usage during deployment with
+binary or ternary weights, training these models still demands substantial
+memory footprints. This is partly because high-precision (i.e., unquantized)
+weight matrices required for straight-through estimation must be maintained
+throughout the whole training process. To address this, we explore the
+potential of directly updating the quantized low-precision weight matrices
+without relying on the straight-through estimator during backpropagation,
+thereby saving memory usage during training. Specifically, we employ a
+stochastic rounding technique to minimize information loss caused by the use of
+low-bit weights throughout training. Experimental results on our
+LLaMA-structured models indicate that (1) training with only low-precision
+weights is feasible even when they are constrained to ternary values, (2)
+extending the bit width to 8 bits results in only a 5% loss degradation
+compared to BitNet b1.58 while offering the potential for reduced memory usage
+during training, and (3) our models can also perform inference using ternary
+weights, showcasing their flexibility in deployment.
+
+
+
+ comment: work in progress
+
+
+
+ + ☆ NLP-ADBench: NLP Anomaly Detection Benchmark SC +
+ +
+ Anomaly detection (AD) is a critical machine learning task with diverse
+applications in web systems, including fraud detection, content moderation, and
+user behavior analysis. Despite its significance, AD in natural language
+processing (NLP) remains underexplored, limiting advancements in detecting
+anomalies in text data such as harmful content, phishing attempts, or spam
+reviews. In this paper, we introduce NLP-ADBench, the most comprehensive
+benchmark for NLP anomaly detection (NLP-AD), comprising eight curated datasets
+and evaluations of nineteen state-of-the-art algorithms. These include three
+end-to-end methods and sixteen two-step algorithms that apply traditional
+anomaly detection techniques to language embeddings generated by
+bert-base-uncased and OpenAI's text-embedding-3-large models.
+ Our results reveal critical insights and future directions for NLP-AD.
+Notably, no single model excels across all datasets, highlighting the need for
+automated model selection. Moreover, two-step methods leveraging
+transformer-based embeddings consistently outperform specialized end-to-end
+approaches, with OpenAI embeddings demonstrating superior performance over BERT
+embeddings. By releasing NLP-ADBench at
+https://github.com/USC-FORTIS/NLP-ADBench, we provide a standardized framework
+for evaluating NLP-AD methods, fostering the development of innovative
+approaches. This work fills a crucial gap in the field and establishes a
+foundation for advancing NLP anomaly detection, particularly in the context of
+improving the safety and reliability of web-based systems.
+
+
+
+ comment: The project is available at https://github.com/USC-FORTIS/NLP-ADBench
+
+
+
+ + ☆ Foundation Models for Low-Resource Language Education (Vision Paper) +
+
+
+
+
+
+
+
+ Zhaojun Ding, Zhengliang Liu, Hanqi Jiang, Yizhu Gao, Xiaoming Zhai, Tianming Liu, Ninghao Liu
+
+
+ Recent studies show that large language models (LLMs) are powerful tools for
+working with natural language, bringing advances in many areas of computational
+linguistics. However, these models face challenges when applied to low-resource
+languages due to limited training data and difficulty in understanding cultural
+nuances. Research is now focusing on multilingual models to improve LLM
+performance for these languages. Education in these languages also struggles
+with a lack of resources and qualified teachers, particularly in underdeveloped
+regions. Here, LLMs can be transformative, supporting innovative methods like
+community-driven learning and digital platforms. This paper discusses how LLMs
+could enhance education for low-resource languages, emphasizing practical
+applications and benefits.
+
+
+
+
+ + ☆ Ltri-LLM: Streaming Long Context Inference for LLMs with Training-Free + Dynamic Triangular Attention Pattern +
+ +
+ The quadratic computational complexity of the attention mechanism in current
+Large Language Models (LLMs) renders inference with long contexts prohibitively
+expensive. To address this challenge, various approaches aim to retain critical
+portions of the context to optimally approximate Full Attention (FA) through
+Key-Value (KV) compression or Sparse Attention (SA), enabling the processing of
+virtually unlimited text lengths in a streaming manner. However, these methods
+struggle to achieve performance levels comparable to FA, particularly in
+retrieval tasks. In this paper, our analysis of attention head patterns reveals
+that LLMs' attention distributions show strong local correlations, naturally
+reflecting a chunking mechanism for input context. We propose Ltri-LLM
+framework, which divides KVs into spans, stores them in an offline index, and
+retrieves the relevant KVs into memory for various queries. Experimental
+results on popular long text benchmarks show that Ltri-LLM can achieve
+performance close to FA while maintaining efficient, streaming-based inference.
+
+
+
+
+ + ☆ ChatNVD: Advancing Cybersecurity Vulnerability Assessment with Large + Language Models +
+ +
+ The increasing frequency and sophistication of cybersecurity vulnerabilities
+in software systems underscore the urgent need for robust and effective methods
+of vulnerability assessment. However, existing approaches often rely on highly
+technical and abstract frameworks, which hinders understanding and increases
+the likelihood of exploitation, resulting in severe cyberattacks. Given the
+growing adoption of Large Language Models (LLMs) across diverse domains, this
+paper explores their potential application in cybersecurity, specifically for
+enhancing the assessment of software vulnerabilities. We propose ChatNVD, an
+LLM-based cybersecurity vulnerability assessment tool leveraging the National
+Vulnerability Database (NVD) to provide context-rich insights and streamline
+vulnerability analysis for cybersecurity professionals, developers, and
+non-technical users. We develop three variants of ChatNVD, utilizing three
+prominent LLMs: GPT-4o mini by OpenAI, Llama 3 by Meta, and Gemini 1.5 Pro by
+Google. To evaluate their efficacy, we conduct a comparative analysis of these
+models using a comprehensive questionnaire comprising common security
+vulnerability questions, assessing their accuracy in identifying and analyzing
+software vulnerabilities. This study provides valuable insights into the
+potential of LLMs to address critical challenges in understanding and
+mitigation of software vulnerabilities.
+
+
+
+
+ + ☆ Question Answering for Decisionmaking in Green Building Design: A + Multimodal Data Reasoning Method Driven by Large Language Models +
+ +
+ In recent years, the critical role of green buildings in addressing energy
+consumption and environmental issues has become widely acknowledged. Research
+indicates that over 40% of potential energy savings can be achieved during the
+early design stage. Therefore, decision-making in green building design (DGBD),
+which is based on modeling and performance simulation, is crucial for reducing
+building energy costs. However, the field of green building encompasses a broad
+range of specialized knowledge, which involves significant learning costs and
+results in low decision-making efficiency. Many studies have already applied
+artificial intelligence (AI) methods to this field. Based on previous research,
+this study innovatively integrates large language models with DGBD, creating
+GreenQA, a question answering framework for multimodal data reasoning.
+Utilizing Retrieval Augmented Generation, Chain of Thought, and Function Call
+methods, GreenQA enables multimodal question answering, including weather data
+analysis and visualization, retrieval of green building cases, and knowledge
+query. Additionally, this study conducted a user survey using the GreenQA web
+platform. The results showed that 96% of users believed the platform helped
+improve design efficiency. This study not only effectively supports DGBD but
+also provides inspiration for AI-assisted design.
+
+
+
+ comment: Published at Association for Computer Aided Design in Architecture
+ (ACADIA) 2024
+
+
+
+ + ☆ BESSTIE: A Benchmark for Sentiment and Sarcasm Classification for + Varieties of English +
+ +
+ Despite large language models (LLMs) being known to exhibit bias against
+non-mainstream varieties, there are no known labeled datasets for sentiment
+analysis of English. To address this gap, we introduce BESSTIE, a benchmark for
+sentiment and sarcasm classification for three varieties of English: Australian
+(en-AU), Indian (en-IN), and British (en-UK). Using web-based content from two
+domains, namely, Google Place reviews and Reddit comments, we collect datasets
+for these language varieties using two methods: location-based and topic-based
+filtering. Native speakers of the language varieties manually annotate the
+datasets with sentiment and sarcasm labels. Subsequently, we fine-tune nine
+large language models (LLMs) (representing a range of encoder/decoder and
+mono/multilingual models) on these datasets, and evaluate their performance on
+the two tasks. Our results reveal that the models consistently perform better
+on inner-circle varieties (i.e., en-AU and en-UK), with significant performance
+drops for en-IN, particularly in sarcasm detection. We also report challenges
+in cross-variety generalisation, highlighting the need for language
+variety-specific datasets such as ours. BESSTIE promises to be a useful
+evaluative benchmark for future research in equitable LLMs, specifically in
+terms of language varieties. The BESSTIE datasets, code, and models are
+currently available on request, while the paper is under review. Please email
+aditya.joshi@unsw.edu.au.
+
+
+
+ comment: 10 pages, 7 figures, under review
+
+
+
+ + ☆ NoLoR: An ASR-Based Framework for Expedited Endangered Language + Documentation with Neo-Aramaic as a Case Study +
+ +
+ The documentation of the Neo-Aramaic dialects before their extinction has
+been described as the most urgent task in all of Semitology today. The death of
+this language will be an unfathomable loss to the descendents of the indigenous
+speakers of Aramaic, now predominantly diasporic after forced displacement due
+to violence. This paper develops an ASR model to expedite the documentation of
+this endangered language and generalizes the strategy in a new framework we
+call NoLoR.
+
+
+
+
+ + ☆ Transformers Struggle to Learn to Search +
+
+
+
+
+
+
+
+ Abulhair Saparov, Srushti Pawar, Shreyas Pimpalgaonkar, Nitish Joshi, Richard Yuanzhe Pang, Vishakh Padmakumar, Seyed Mehran Kazemi, Najoung Kim, He He
+
+
+ Search is an ability foundational in many important tasks, and recent studies
+have shown that large language models (LLMs) struggle to perform search
+robustly. It is unknown whether this inability is due to a lack of data,
+insufficient model parameters, or fundamental limitations of the transformer
+architecture. In this work, we use the foundational graph connectivity problem
+as a testbed to generate effectively limitless high-coverage data to train
+small transformers and test whether they can learn to perform search. We find
+that, when given the right training distribution, the transformer is able to
+learn to search.
+ We analyze the algorithm that the transformer has learned through a novel
+mechanistic interpretability technique that enables us to extract the
+computation graph from the trained model. We find that for each vertex in the
+input graph, transformers compute the set of vertices reachable from that
+vertex. Each layer then progressively expands these sets, allowing the model to
+search over a number of vertices exponential in the number of layers.
+ However, we find that as the input graph size increases, the transformer has
+greater difficulty in learning the task. This difficulty is not resolved even
+as the number of parameters is increased, suggesting that increasing model
+scale will not lead to robust search abilities. We also find that performing
+search in-context (i.e., chain-of-thought) does not resolve this inability to
+learn to search on larger graphs.
+
+
+
+
+ + ☆ Privacy-Preserving Retrieval Augmented Generation with Differential + Privacy +
+ +
+ With the recent remarkable advancement of large language models (LLMs), there
+has been a growing interest in utilizing them in the domains with highly
+sensitive data that lies outside their training data. For this purpose,
+retrieval augmented generation (RAG) is particularly effective -- it assists
+LLMs by directly providing relevant information from the external knowledge
+sources. However, without extra privacy safeguards, RAG outputs risk leaking
+sensitive information from the external data source. In this work, we explore
+RAG under differential privacy (DP), a formal guarantee of data privacy. The
+main challenge with differentially private RAG is how to generate long accurate
+answers within a moderate privacy budget. We address this by proposing an
+algorithm that smartly spends privacy budget only for the tokens that require
+the sensitive information and uses the non-private LLM for other tokens. Our
+extensive empirical evaluations reveal that our algorithm outperforms the
+non-RAG baseline under a reasonable privacy budget of $\epsilon\approx 10$
+across different models and datasets.
+
+
+
+
+ + ☆ LLM-Align: Utilizing Large Language Models for Entity Alignment in + Knowledge Graphs +
+ +
+ Entity Alignment (EA) seeks to identify and match corresponding entities
+across different Knowledge Graphs (KGs), playing a crucial role in knowledge
+fusion and integration. Embedding-based entity alignment (EA) has recently
+gained considerable attention, resulting in the emergence of many innovative
+approaches. Initially, these approaches concentrated on learning entity
+embeddings based on the structural features of knowledge graphs (KGs) as
+defined by relation triples. Subsequent methods have integrated entities' names
+and attributes as supplementary information to improve the embeddings used for
+EA. However, existing methods lack a deep semantic understanding of entity
+attributes and relations. In this paper, we propose a Large Language Model
+(LLM) based Entity Alignment method, LLM-Align, which explores the
+instruction-following and zero-shot capabilities of Large Language Models to
+infer alignments of entities. LLM-Align uses heuristic methods to select
+important attributes and relations of entities, and then feeds the selected
+triples of entities to an LLM to infer the alignment results. To guarantee the
+quality of alignment results, we design a multi-round voting mechanism to
+mitigate the hallucination and positional bias issues that occur with LLMs.
+Experiments on three EA datasets, demonstrating that our approach achieves
+state-of-the-art performance compared to existing EA methods.
+
+
+
+
+ + ♻ ☆ A Practitioner's Guide to Continual Multimodal Pretraining NeurIPS + 2024 +
+
+
+
+
+
+
+
+ Karsten Roth, Vishaal Udandarao, Sebastian Dziadzio, Ameya Prabhu, Mehdi Cherti, Oriol Vinyals, Olivier Hénaff, Samuel Albanie, Matthias Bethge, Zeynep Akata
+
+
+ Multimodal foundation models serve numerous applications at the intersection
+of vision and language. Still, despite being pretrained on extensive data, they
+become outdated over time. To keep models updated, research into continual
+pretraining mainly explores scenarios with either (1) infrequent,
+indiscriminate updates on large-scale new data, or (2) frequent, sample-level
+updates. However, practical model deployment often operates in the gap between
+these two limit cases, as real-world applications often demand adaptation to
+specific subdomains, tasks or concepts -- spread over the entire, varying life
+cycle of a model. In this work, we complement current perspectives on continual
+pretraining through a research test bed as well as provide comprehensive
+guidance for effective continual model updates in such scenarios. We first
+introduce FoMo-in-Flux, a continual multimodal pretraining benchmark with
+realistic compute constraints and practical deployment requirements,
+constructed over 63 datasets with diverse visual and semantic coverage. Using
+FoMo-in-Flux, we explore the complex landscape of practical continual
+pretraining through multiple perspectives: (1) A data-centric investigation of
+data mixtures and stream orderings that emulate real-world deployment
+situations, (2) a method-centric investigation ranging from simple fine-tuning
+and traditional continual learning strategies to parameter-efficient updates
+and model merging, (3) meta learning rate schedules and mechanistic design
+choices, and (4) the influence of model and compute scaling. Together, our
+insights provide a practitioner's guide to continual multimodal pretraining for
+real-world deployment. Our benchmark and code is here:
+https://github.com/ExplainableML/fomo_in_flux.
+
+
+
+ comment: Technical Report. 52 pages. Shorter version published at the NeurIPS
+ 2024 Dataset & Benchmarks track
+
+
+
+ + ♻ ☆ Is Your Paper Being Reviewed by an LLM? Investigating AI Text + Detectability in Peer Review +
+ +
+ Peer review is a critical process for ensuring the integrity of published
+scientific research. Confidence in this process is predicated on the assumption
+that experts in the relevant domain give careful consideration to the merits of
+manuscripts which are submitted for publication. With the recent rapid
+advancements in the linguistic capabilities of large language models (LLMs), a
+new potential risk to the peer review process is that negligent reviewers will
+rely on LLMs to perform the often time consuming process of reviewing a paper.
+In this study, we investigate the ability of existing AI text detection
+algorithms to distinguish between peer reviews written by humans and different
+state-of-the-art LLMs. Our analysis shows that existing approaches fail to
+identify many GPT-4o written reviews without also producing a high number of
+false positive classifications. To address this deficiency, we propose a new
+detection approach which surpasses existing methods in the identification of
+GPT-4o written peer reviews at low levels of false positive classifications.
+Our work reveals the difficulty of accurately identifying AI-generated text at
+the individual review level, highlighting the urgent need for new tools and
+methods to detect this type of unethical application of generative AI.
+
+
+
+
+ + ♻ ☆ Unlocking State-Tracking in Linear RNNs Through Negative Eigenvalues +
+
+
+
+
+
+
+
+ Riccardo Grazzi, Julien Siems, Jörg K. H. Franke, Arber Zela, Frank Hutter, Massimiliano Pontil
+
+
+ Linear Recurrent Neural Networks (LRNNs) such as Mamba, RWKV, GLA, mLSTM, and
+DeltaNet have emerged as efficient alternatives to Transformers in large
+language modeling, offering linear scaling with sequence length and improved
+training efficiency. However, LRNNs struggle to perform state-tracking which
+may impair performance in tasks such as code evaluation or tracking a chess
+game. Even parity, the simplest state-tracking task, which non-linear RNNs like
+LSTM handle effectively, cannot be solved by current LRNNs. Recently, Sarrof et
+al. (2024) demonstrated that the failure of LRNNs like Mamba to solve parity
+stems from restricting the value range of their diagonal state-transition
+matrices to $[0, 1]$ and that incorporating negative values can resolve this
+issue. We extend this result to non-diagonal LRNNs, which have recently shown
+promise in models such as DeltaNet. We prove that finite precision LRNNs with
+state-transition matrices having only positive eigenvalues cannot solve parity,
+while complex eigenvalues are needed to count modulo $3$. Notably, we also
+prove that LRNNs can learn any regular language when their state-transition
+matrices are products of identity minus vector outer product matrices, each
+with eigenvalues in the range $[-1, 1]$. Our empirical results confirm that
+extending the eigenvalue range of models like Mamba and DeltaNet to include
+negative values not only enables them to solve parity but consistently improves
+their performance on state-tracking tasks. Furthermore, pre-training LRNNs with
+an extended eigenvalue range for language modeling achieves comparable
+performance and stability while showing promise on code and math data. Our work
+enhances the expressivity of modern LRNNs, broadening their applicability
+without changing the cost of training or inference.
+
+
+
+ comment: Main changes: Correction to Theorem 1 and 2 (we excluded from the
+ only if condition complex eigenvalues with modulus strictly less than one).
+ Correction to point 3 of Proposition 3
+
+
+
+ + ♻ ☆ Random Tree Model of Meaningful Memory +
+
+
+
+
+
+
+
+ Weishun Zhong, Tankut Can, Antonis Georgiou, Ilya Shnayderman, Mikhail Katkov, Misha Tsodyks
+
+
+ Traditional studies of memory for meaningful narratives focus on specific
+stories and their semantic structures but do not address common quantitative
+features of recall across different narratives. We introduce a statistical
+ensemble of random trees to represent narratives as hierarchies of key points,
+where each node is a compressed representation of its descendant leaves, which
+are the original narrative segments. Recall is modeled as constrained by
+working memory capacity from this hierarchical structure. Our analytical
+solution aligns with observations from large-scale narrative recall
+experiments. Specifically, our model explains that (1) average recall length
+increases sublinearly with narrative length, and (2) individuals summarize
+increasingly longer narrative segments in each recall sentence. Additionally,
+the theory predicts that for sufficiently long narratives, a universal,
+scale-invariant limit emerges, where the fraction of a narrative summarized by
+a single recall sentence follows a distribution independent of narrative
+length.
+
+
+
+ comment: 16 pages, 4 figures
+
+
+
+ + ♻ ☆ MultiTrust: A Comprehensive Benchmark Towards Trustworthy Multimodal + Large Language Models +
+
+
+
+
+
+
+
+ Yichi Zhang, Yao Huang, Yitong Sun, Chang Liu, Zhe Zhao, Zhengwei Fang, Yifan Wang, Huanran Chen, Xiao Yang, Xingxing Wei, Hang Su, Yinpeng Dong, Jun Zhu
+
+
+ Despite the superior capabilities of Multimodal Large Language Models (MLLMs)
+across diverse tasks, they still face significant trustworthiness challenges.
+Yet, current literature on the assessment of trustworthy MLLMs remains limited,
+lacking a holistic evaluation to offer thorough insights into future
+improvements. In this work, we establish MultiTrust, the first comprehensive
+and unified benchmark on the trustworthiness of MLLMs across five primary
+aspects: truthfulness, safety, robustness, fairness, and privacy. Our benchmark
+employs a rigorous evaluation strategy that addresses both multimodal risks and
+cross-modal impacts, encompassing 32 diverse tasks with self-curated datasets.
+Extensive experiments with 21 modern MLLMs reveal some previously unexplored
+trustworthiness issues and risks, highlighting the complexities introduced by
+the multimodality and underscoring the necessity for advanced methodologies to
+enhance their reliability. For instance, typical proprietary models still
+struggle with the perception of visually confusing images and are vulnerable to
+multimodal jailbreaking and adversarial attacks; MLLMs are more inclined to
+disclose privacy in text and reveal ideological and cultural biases even when
+paired with irrelevant images in inference, indicating that the multimodality
+amplifies the internal risks from base LLMs. Additionally, we release a
+scalable toolbox for standardized trustworthiness research, aiming to
+facilitate future advancements in this important field. Code and resources are
+publicly available at: https://multi-trust.github.io/.
+
+
+
+ comment: 100 pages, 84 figures, 33 tables
+
+
+
+ + ♻ ☆ Sensitive Content Classification in Social Media: A Holistic Resource + and Evaluation +
+
+
+
+
+
+
+
+ Dimosthenis Antypas, Indira Sen, Carla Perez-Almendros, Jose Camacho-Collados, Francesco Barbieri
+
+
+ The detection of sensitive content in large datasets is crucial for ensuring
+that shared and analysed data is free from harmful material. However, current
+moderation tools, such as external APIs, suffer from limitations in
+customisation, accuracy across diverse sensitive categories, and privacy
+concerns. Additionally, existing datasets and open-source models focus
+predominantly on toxic language, leaving gaps in detecting other sensitive
+categories such as substance abuse or self-harm. In this paper, we put forward
+a unified dataset tailored for social media content moderation across six
+sensitive categories: conflictual language, profanity, sexually explicit
+material, drug-related content, self-harm, and spam. By collecting and
+annotating data with consistent retrieval strategies and guidelines, we address
+the shortcomings of previous focalised research. Our analysis demonstrates that
+fine-tuning large language models (LLMs) on this novel dataset yields
+significant improvements in detection performance compared to open
+off-the-shelf models such as LLaMA, and even proprietary OpenAI models, which
+underperform by 10-15% overall. This limitation is even more pronounced on
+popular moderation APIs, which cannot be easily tailored to specific sensitive
+content categories, among others.
+
+
+
+
+ + ♻ ☆ An Evolved Universal Transformer Memory +
+ +
+ Prior methods propose to offset the escalating costs of modern foundation
+models by dropping specific parts of their contexts with hand-designed rules,
+while attempting to preserve their original performance. We overcome this
+trade-off with Neural Attention Memory Models (NAMMs), introducing a learned
+network for memory management that improves both the performance and efficiency
+of transformers. We evolve NAMMs atop pre-trained transformers to provide
+different latent contexts focusing on the most relevant information for
+individual layers and attention heads. NAMMs are universally applicable to any
+model using self-attention as they condition exclusively on the values in the
+produced attention matrices. Learning NAMMs on a small set of problems, we
+achieve substantial performance improvements across multiple long-context
+benchmarks while cutting the model's input contexts up to a fraction of the
+original sizes. We show the generality of our conditioning enables zero-shot
+transfer of NAMMs trained only on language to entirely new transformer
+architectures even across input modalities, with their benefits carrying over
+to vision and reinforcement learning.
+
+
+
+ comment: Preprint, under submission. Source code is available at
+ https://github.com/SakanaAI/evo-memory
+
+
+
+ + ♻ ☆ Hallucination Detection in LLMs: Fast and Memory-Efficient Fine-Tuned + Models +
+ +
+ Uncertainty estimation is a necessary component when implementing AI in
+high-risk settings, such as autonomous cars, medicine, or insurances. Large
+Language Models (LLMs) have seen a surge in popularity in recent years, but
+they are subject to hallucinations, which may cause serious harm in high-risk
+settings. Despite their success, LLMs are expensive to train and run: they need
+a large amount of computations and memory, preventing the use of ensembling
+methods in practice. In this work, we present a novel method that allows for
+fast and memory-friendly training of LLM ensembles. We show that the resulting
+ensembles can detect hallucinations and are a viable approach in practice as
+only one GPU is needed for training and inference.
+
+
+
+ comment: 6 pages, 3 figures
+
+
+
+ + ♻ ☆ LLMs May Perform MCQA by Selecting the Least Incorrect Option COLING 2025 +
+ +
+ In the field of NLP, Large Language Models (LLMs) have markedly enhanced
+performance across a variety of tasks. However, the comprehensive evaluation of
+LLMs remains an inevitable challenge for the community. Recently, the adoption
+of Multiple Choice Question Answering (MCQA) as a benchmark for assessing LLMs
+has gained considerable traction. However, concerns regarding the robustness of
+this evaluative method persist. Building upon previous discussions on the issue
+of \textit{variability}, we reveal an additional dimension of concern: LLMs may
+perform MCQA by selecting the least incorrect option rather than distinctly
+correct. This observation suggests that LLMs might regard multiple options as
+correct, which could undermine the reliability of MCQA as a metric for
+evaluating LLMs. To address this challenge, we introduce an enhanced dataset
+augmentation method for MCQA, termed MCQA+, to provide a more accurate
+reflection of the model performance, thereby highlighting the necessity for
+more sophisticated evaluation mechanisms in the assessment of LLM capabilities.
+
+
+
+ comment: COLING 2025
+
+
+
+ + ♻ ☆ Densing Law of LLMs +
+
+
+
+
+
+
+
+ Chaojun Xiao, Jie Cai, Weilin Zhao, Guoyang Zeng, Biyuan Lin, Jie Zhou, Zhi Zheng, Xu Han, Zhiyuan Liu, Maosong Sun
+
+
+ Large Language Models (LLMs) have emerged as a milestone in artificial
+intelligence, and their performance can improve as the model size increases.
+However, this scaling brings great challenges to training and inference
+efficiency, particularly for deploying LLMs in resource-constrained
+environments, and the scaling trend is becoming increasingly unsustainable.
+This paper introduces the concept of ``\textit{capacity density}'' as a new
+metric to evaluate the quality of the LLMs across different scales and
+describes the trend of LLMs in terms of both effectiveness and efficiency. To
+calculate the capacity density of a given target LLM, we first introduce a set
+of reference models and develop a scaling law to predict the downstream
+performance of these reference models based on their parameter sizes. We then
+define the \textit{effective parameter size} of the target LLM as the parameter
+size required by a reference model to achieve equivalent performance, and
+formalize the capacity density as the ratio of the effective parameter size to
+the actual parameter size of the target LLM. Capacity density provides a
+unified framework for assessing both model effectiveness and efficiency. Our
+further analysis of recent open-source base LLMs reveals an empirical law (the
+densing law)that the capacity density of LLMs grows exponentially over time.
+More specifically, using some widely used benchmarks for evaluation, the
+capacity density of LLMs doubles approximately every three months. The law
+provides new perspectives to guide future LLM development, emphasizing the
+importance of improving capacity density to achieve optimal results with
+minimal computational overhead.
+
+
+
+
+ + ♻ ☆ Docling Technical Report AAAI 25 +
+
+
+
+
+
+
+
+ Nikolaos Livathinos, Christoph Auer, Maksym Lysak, Ahmed Nassar, Michele Dolfi, Panos Vagenas, Cesar Berrospi Ramis, Matteo Omenetti, Kasper Dinkla, Yusik Kim, Shubham Gupta, Rafael Teixeira de Lima, Valery Weber, Lucas Morin, Ingmar Meijer, Viktor Kuropiatnyk, Peter W. J. Staar
+
+
+ We introduce Docling, an easy-to-use, self-contained, MIT-licensed,
+open-source toolkit for document conversion, that can parse several types of
+popular document formats into a unified, richly structured representation. It
+is powered by state-of-the-art specialized AI models for layout analysis
+(DocLayNet) and table structure recognition (TableFormer), and runs efficiently
+on commodity hardware in a small resource budget. Docling is released as a
+Python package and can be used as a Python API or as a CLI tool. Docling's
+modular architecture and efficient document representation %, known as
+DoclingDocument, make it easy to implement extensions, new features, models,
+and customizations. Docling has been already integrated in other popular
+open-source frameworks (e.g., LlamaIndex, LangChain, spaCy), making it a
+natural fit for the processing of documents and the development of high-end
+applications. The open-source community has fully engaged in using, promoting,
+and developing for Docling, which gathered 10k stars on GitHub in less than a
+month and was reported as the No. 1 trending repository in GitHub worldwide in
+November 2024.
+
+
+
+ comment: Submitted to AAAI 25: Workshop on Open-Source AI for Mainstream Use
+
+
+
+ + ♻ ☆ Endless Jailbreaks with Bijection Learning +
+ +
+ Despite extensive safety measures, LLMs are vulnerable to adversarial inputs,
+or jailbreaks, which can elicit unsafe behaviors. In this work, we introduce
+bijection learning, a powerful attack algorithm which automatically fuzzes LLMs
+for safety vulnerabilities using randomly-generated encodings whose complexity
+can be tightly controlled. We leverage in-context learning to teach models
+bijective encodings, pass encoded queries to the model to bypass built-in
+safety mechanisms, and finally decode responses back into English. Our attack
+is extremely effective on a wide range of frontier language models. Moreover,
+by controlling complexity parameters such as number of key-value mappings in
+the encodings, we find a close relationship between the capability level of the
+attacked LLM and the average complexity of the most effective bijection
+attacks. Our work highlights that new vulnerabilities in frontier models can
+emerge with scale: more capable models are more severely jailbroken by
+bijection attacks.
+
+
+
+
+ + ♻ ☆ Enhancing Zero-shot Chain of Thought Prompting via Uncertainty-Guided + Strategy Selection COLING 2025 +
+
+
+
+
+
+
+
+ Shanu Kumar, Saish Mendke, Karody Lubna Abdul Rahman, Santosh Kurasa, Parag Agrawal, Sandipan Dandapat
+
+
+ Chain-of-thought (CoT) prompting has significantly enhanced the capability of
+large language models (LLMs) by structuring their reasoning processes. However,
+existing methods face critical limitations: handcrafted demonstrations require
+extensive human expertise, while trigger phrases are prone to inaccuracies. In
+this paper, we propose the Zero-shot Uncertainty-based Selection (ZEUS) method,
+a novel approach that improves CoT prompting by utilizing uncertainty estimates
+to select effective demonstrations without needing access to model parameters.
+Unlike traditional methods, ZEUS offers high sensitivity in distinguishing
+between helpful and ineffective questions, ensuring more precise and reliable
+selection. Our extensive evaluation shows that ZEUS consistently outperforms
+existing CoT strategies across four challenging reasoning benchmarks,
+demonstrating its robustness and scalability.
+
+
+
+ comment: Accepted in COLING 2025
+
+
+
+ + ♻ ☆ Acquired TASTE: Multimodal Stance Detection with Textual and Structural + Embeddings COLING +
+ +
+ Stance detection plays a pivotal role in enabling an extensive range of
+downstream applications, from discourse parsing to tracing the spread of fake
+news and the denial of scientific facts. While most stance classification
+models rely on textual representation of the utterance in question, prior work
+has demonstrated the importance of the conversational context in stance
+detection. In this work we introduce TASTE -- a multimodal architecture for
+stance detection that harmoniously fuses Transformer-based content embedding
+with unsupervised structural embedding. Through the fine-tuning of a pretrained
+transformer and the amalgamation with social embedding via a Gated Residual
+Network (GRN) layer, our model adeptly captures the complex interplay between
+content and conversational structure in determining stance. TASTE achieves
+state-of-the-art results on common benchmarks, significantly outperforming an
+array of strong baselines. Comparative evaluations underscore the benefits of
+social grounding -- emphasizing the criticality of concurrently harnessing both
+content and structure for enhanced stance detection.
+
+
+
+ comment: The modified camera ready version will be published in January 2025
+ at COLING
+
+
+
+ + ♻ ☆ On the Proper Treatment of Tokenization in Psycholinguistics EMNLP 2024 +
+
+
+
+
+
+
+
+ Mario Giulianelli, Luca Malagutti, Juan Luis Gastaldi, Brian DuSell, Tim Vieira, Ryan Cotterell
+
+
+ Language models are widely used in computational psycholinguistics to test
+theories that relate the negative log probability (the surprisal) of a region
+of interest (a substring of characters) under a language model to its cognitive
+cost experienced by readers, as operationalized, for example, by gaze duration
+on the region. However, the application of modern language models to
+psycholinguistic studies is complicated by the practice of using tokenization
+as an intermediate step in training a model. Doing so results in a language
+model over token strings rather than one over character strings. Vexingly,
+regions of interest are generally misaligned with these token strings. The
+paper argues that token-level language models should be (approximately)
+marginalized into character-level language models before they are used in
+psycholinguistic studies to compute the surprisal of a region of interest;
+then, the marginalized character-level language model can be used to compute
+the surprisal of an arbitrary character substring, which we term a focal area,
+that the experimenter may wish to use as a predictor. Our proposal of
+marginalizing a token-level model into a character-level one solves this
+misalignment issue independently of the tokenization scheme. Empirically, we
+discover various focal areas whose surprisal is a better psychometric predictor
+than the surprisal of the region of interest itself.
+
+
+
+ comment: Main conference long paper at EMNLP 2024. New version: copy-editing
+ and updated bib
+
+
+
+ + ♻ ☆ Autoformalize Mathematical Statements by Symbolic Equivalence and + Semantic Consistency NeurIPS 2024 +
+ +
+ Autoformalization, the task of automatically translating natural language
+descriptions into a formal language, poses a significant challenge across
+various domains, especially in mathematics. Recent advancements in large
+language models (LLMs) have unveiled their promising capabilities to formalize
+even competition-level math problems. However, we observe a considerable
+discrepancy between pass@1 and pass@k accuracies in LLM-generated
+formalizations. To address this gap, we introduce a novel framework that scores
+and selects the best result from k autoformalization candidates based on two
+complementary self-consistency methods: symbolic equivalence and semantic
+consistency. Elaborately, symbolic equivalence identifies the logical
+homogeneity among autoformalization candidates using automated theorem provers,
+and semantic consistency evaluates the preservation of the original meaning by
+informalizing the candidates and computing the similarity between the
+embeddings of the original and informalized texts. Our extensive experiments on
+the MATH and miniF2F datasets demonstrate that our approach significantly
+enhances autoformalization accuracy, achieving up to 0.22-1.35x relative
+improvements across various LLMs and baseline methods.
+
+
+
+ comment: Published as a conference paper at NeurIPS 2024. Code is available at
+ https://github.com/Miracle-Messi/Isa-AutoFormal
+
+
+
+ + ♻ ☆ COBias and Debias: Minimizing Language Model Pairwise Accuracy Bias via + Nonlinear Integer Programming +
+ +
+ When performing classification tasks with language models, would you prefer
+having only one highly accurate class or having every class deliver reliable
+performance? Obviously, a more balanced accuracy among classes better reflects
+the expectations of the majority of users. Especially for large language models
+(LLMs), the fact that they achieve a fair overall accuracy by in-context
+learning (ICL) obscures a large difference in individual class accuracies. In
+this work, we uncover and tackle language models' imbalance in per-class
+prediction accuracy by reconceptualizing it as the Contextual Oddity Bias
+(COBias), and we are the first to engage nonlinear integer programming (NIP) to
+debias it. Briefly, the proposed COBias metric measures accuracy differences
+among class pairs, with which we reveal the large per-class accuracy
+differences exhibited in LLMs of varied scales and families. Then we propose
+Debiasing as Nonlinear Integer Programming (DNIP) to correct ICL per-class
+probabilities towards lower COBias and higher overall accuracy. Our
+optimization objective is directly based on the evaluation scores by COBias and
+accuracy metrics, which is non-differentiable and solved by the simulated
+annealing metaheuristic. Evaluations on three LLMs across seven NLP
+classification tasks show that DNIP simultaneously achieves significant COBias
+reduction (-27%) and accuracy improvement (+12%) over the conventional ICL
+approach, suggesting that modeling pairwise class accuracy differences is a
+direction in pushing forward more accurate, more reliable LLM predictions.
+
+
+
+
+ + ♻ ☆ DiversityMedQA: Assessing Demographic Biases in Medical Diagnosis using + Large Language Models EMNLP 2024 +
+
+
+
+
+
+
+
+ Rajat Rawat, Hudson McBride, Dhiyaan Nirmal, Rajarshi Ghosh, Jong Moon, Dhruv Alamuri, Sean O'Brien, Kevin Zhu
+
+
+ As large language models (LLMs) gain traction in healthcare, concerns about
+their susceptibility to demographic biases are growing. We introduce
+{DiversityMedQA}, a novel benchmark designed to assess LLM responses to medical
+queries across diverse patient demographics, such as gender and ethnicity. By
+perturbing questions from the MedQA dataset, which comprises medical board exam
+questions, we created a benchmark that captures the nuanced differences in
+medical diagnosis across varying patient profiles. Our findings reveal notable
+discrepancies in model performance when tested against these demographic
+variations. Furthermore, to ensure the perturbations were accurate, we also
+propose a filtering strategy that validates each perturbation. By releasing
+DiversityMedQA, we provide a resource for evaluating and mitigating demographic
+bias in LLM medical diagnoses.
+
+
+
+ comment: Published in NLP4PI @ EMNLP 2024, Accepted to AIM-FM @ NeurIPS 2024
+
+
+
+ + ♻ ☆ Plentiful Jailbreaks with String Compositions NeurIPS +
+ +
+ Large language models (LLMs) remain vulnerable to a slew of adversarial
+attacks and jailbreaking methods. One common approach employed by white-hat
+attackers, or red-teamers, is to process model inputs and outputs using
+string-level obfuscations, which can include leetspeak, rotary ciphers, Base64,
+ASCII, and more. Our work extends these encoding-based attacks by unifying them
+in a framework of invertible string transformations. With invertibility, we can
+devise arbitrary string compositions, defined as sequences of transformations,
+that we can encode and decode end-to-end programmatically. We devise a
+automated best-of-n attack that samples from a combinatorially large number of
+string compositions. Our jailbreaks obtain competitive attack success rates on
+several leading frontier models when evaluated on HarmBench, highlighting that
+encoding-based attacks remain a persistent vulnerability even in advanced LLMs.
+
+
+
+ comment: NeurIPS SoLaR Workshop 2024
+
+
+
+ + ♻ ☆ U-MATH: A University-Level Benchmark for Evaluating Mathematical Skills + in LLMs +
+
+
+
+
+
+
+
+ Konstantin Chernyshev, Vitaliy Polshkov, Ekaterina Artemova, Alex Myasnikov, Vlad Stepanov, Alexei Miasnikov, Sergei Tilga
+
+
+ The current evaluation of mathematical skills in LLMs is limited, as existing
+benchmarks are either relatively small, primarily focus on elementary and
+high-school problems, or lack diversity in topics. Additionally, the inclusion
+of visual elements in tasks remains largely under-explored.
+ To address these gaps, we introduce U-MATH, a novel benchmark of 1,100
+unpublished open-ended university-level problems sourced from teaching
+materials. It is balanced across six core subjects, with 20% of multimodal
+problems. Given the open-ended nature of U-MATH problems, we employ an LLM to
+judge the correctness of generated solutions. To this end, we release
+$\mu$-MATH, a dataset to evaluate the LLMs' capabilities in judging solutions.
+ The evaluation of general domain, math-specific, and multimodal LLMs
+highlights the challenges presented by U-MATH. Our findings reveal that LLMs
+achieve a maximum accuracy of only 63% on text-based tasks, with even lower 45%
+on visual problems. The solution assessment proves challenging for LLMs, with
+the best LLM judge having an F1-score of 80% on $\mu$-MATH.
+
+
+
+
+ + ♻ ☆ LongVALE: Vision-Audio-Language-Event Benchmark Towards Time-Aware + Omni-Modal Perception of Long Videos +
+ +
+ Despite impressive advancements in video understanding, most efforts remain
+limited to coarse-grained or visual-only video tasks. However, real-world
+videos encompass omni-modal information (vision, audio, and speech) with a
+series of events forming a cohesive storyline. The lack of multi-modal video
+data with fine-grained event annotations and the high cost of manual labeling
+are major obstacles to comprehensive omni-modality video perception. To address
+this gap, we propose an automatic pipeline consisting of high-quality
+multi-modal video filtering, semantically coherent omni-modal event boundary
+detection, and cross-modal correlation-aware event captioning. In this way, we
+present LongVALE, the first-ever Vision-Audio-Language Event understanding
+benchmark comprising 105K omni-modal events with precise temporal boundaries
+and detailed relation-aware captions within 8.4K high-quality long videos.
+Further, we build a baseline that leverages LongVALE to enable video large
+language models (LLMs) for omni-modality fine-grained temporal video
+understanding for the first time. Extensive experiments demonstrate the
+effectiveness and great potential of LongVALE in advancing comprehensive
+multi-modal video understanding.
+
+
+
+ comment: 18 pages, 15 figures
+
+
+
+ + ♻ ☆ OPT-Tree: Speculative Decoding with Adaptive Draft Tree Structure ACL +
+
+
+
+
+
+
+
+ Jikai Wang, Yi Su, Juntao Li, Qingrong Xia, Zi Ye, Xinyu Duan, Zhefeng Wang, Min Zhang
+
+
+ Autoregressive language models demonstrate excellent performance in various
+scenarios. However, the inference efficiency is limited by its
+one-step-one-word generation mode, which has become a pressing problem recently
+as the models become increasingly larger. Speculative decoding employs a "draft
+and then verify" mechanism to allow multiple tokens to be generated in one
+step, realizing lossless acceleration. Existing methods mainly adopt fixed
+heuristic draft structures, which fail to adapt to different situations to
+maximize the acceptance length during verification. To alleviate this dilemma,
+we proposed OPT-Tree, an algorithm to construct adaptive and scalable draft
+trees. It searches the optimal tree structure that maximizes the mathematical
+expectation of the acceptance length in each decoding step. Experimental
+results reveal that OPT-Tree outperforms the existing draft structures and
+achieves a speed-up ratio of up to 3.2 compared with autoregressive decoding.
+If the draft model is powerful enough and the node budget is sufficient, it can
+generate more than ten tokens in a single step. Our code is available at
+https://github.com/Jikai0Wang/OPT-Tree.
+
+
+
+ comment: Accepted at TACL; pre-MIT Press publication version
+
+
+
+ + ♻ ☆ PsycoLLM: Enhancing LLM for Psychological Understanding and Evaluation ACL +
+
+
+
+
+
+
+
+ Jinpeng Hu, Tengteng Dong, Luo Gang, Hui Ma, Peng Zou, Xiao Sun, Dan Guo, Xun Yang, Meng Wang
+
+
+ Mental health has attracted substantial attention in recent years and LLM can
+be an effective technology for alleviating this problem owing to its capability
+in text understanding and dialogue. However, existing research in this domain
+often suffers from limitations, such as training on datasets lacking crucial
+prior knowledge and evidence, and the absence of comprehensive evaluation
+methods. In this paper, we propose a specialized psychological large language
+model (LLM), named PsycoLLM, trained on a proposed high-quality psychological
+dataset, including single-turn QA, multi-turn dialogues and knowledge-based QA.
+Specifically, we construct multi-turn dialogues through a three-step pipeline
+comprising multi-turn QA generation, evidence judgment, and dialogue
+refinement. We augment this process with real-world psychological case
+backgrounds extracted from online platforms, enhancing the relevance and
+applicability of the generated data. Additionally, to compare the performance
+of PsycoLLM with other LLMs, we develop a comprehensive psychological benchmark
+based on authoritative psychological counseling examinations in China, which
+includes assessments of professional ethics, theoretical proficiency, and case
+analysis. The experimental results on the benchmark illustrate the
+effectiveness of PsycoLLM, which demonstrates superior performance compared to
+other LLMs.
+
+
+
+ comment: Accepted by IEEE Transactions on Computational Social Systems.
+ https://github.com/MACLAB-HFUT/PsycoLLM
+
+
+
+ + ♻ ☆ CUE-M: Contextual Understanding and Enhanced Search with Multimodal + Large Language Model +
+
+
+
+
+
+
+
+ Dongyoung Go, Taesun Whang, Chanhee Lee, Hwa-Yeon Kim, Sunghoon Park, Seunghwan Ji, Jinho Kim, Dongchan Kim, Young-Bum Kim
+
+
+ The integration of Retrieval-Augmented Generation (RAG) with Multimodal Large
+Language Models (MLLMs) has revolutionized information retrieval and expanded
+the practical applications of AI. However, current systems struggle in
+accurately interpreting user intent, employing diverse retrieval strategies,
+and effectively filtering unintended or inappropriate responses, limiting their
+effectiveness. This paper introduces Contextual Understanding and Enhanced
+Search with MLLM (CUE-M), a novel multimodal search framework that addresses
+these challenges through a multi-stage pipeline comprising image context
+enrichment, intent refinement, contextual query generation, external API
+integration, and relevance-based filtering. CUE-M incorporates a robust
+filtering pipeline combining image-based, text-based, and multimodal
+classifiers, dynamically adapting to instance- and category-specific concern
+defined by organizational policies. Evaluations on a multimodal Q&A dataset and
+a public safety benchmark demonstrate that CUE-M outperforms baselines in
+accuracy, knowledge integration, and safety, advancing the capabilities of
+multimodal retrieval systems.
+
+
+
+ comment: Preprint. Under review
+
+
+
+ + ♻ ☆ DRS: Deep Question Reformulation With Structured Output +
+ +
+ Question answering represents a core capability of large language models
+(LLMs). However, when individuals encounter unfamiliar knowledge in texts, they
+often formulate questions that the text itself cannot answer due to
+insufficient understanding of the underlying information. Recent studies reveal
+that while LLMs can detect unanswerable questions, they struggle to assist
+users in reformulating these questions. Even advanced models like GPT-3.5
+demonstrate limited effectiveness in this regard. To address this limitation,
+we propose DRS: Deep Question Reformulation with Structured Output, a novel
+zero-shot method aimed at enhancing LLMs ability to assist users in
+reformulating questions to extract relevant information from new documents. DRS
+combines the strengths of LLMs with a DFS-based algorithm to iteratively
+explore potential entity combinations and constrain outputs using predefined
+entities. This structured approach significantly enhances the reformulation
+capabilities of LLMs. Comprehensive experimental evaluations demonstrate that
+DRS improves the reformulation accuracy of GPT-3.5 from 23.03% to 70.42%, while
+also enhancing the performance of open-source models, such as Gemma2-9B, from
+26.35% to 56.75%.
+
+
+
+
+ + ♻ ☆ Scaling Inference-Time Search with Vision Value Model for Improved + Visual Comprehension +
+
+
+
+
+
+
+
+ Xiyao Wang, Zhengyuan Yang, Linjie Li, Hongjin Lu, Yuancheng Xu, Chung-Ching Lin, Kevin Lin, Furong Huang, Lijuan Wang
+
+
+ Despite significant advancements in vision-language models (VLMs), there
+lacks effective approaches to enhance response quality by scaling
+inference-time computation. This capability is known to be a core step towards
+the self-improving models in recent large language model studies. In this
+paper, we present Vision Value Model (VisVM) that can guide VLM inference-time
+search to generate responses with better visual comprehension. Specifically,
+VisVM not only evaluates the generated sentence quality in the current search
+step, but also anticipates the quality of subsequent sentences that may result
+from the current step, thus providing a long-term value. In this way, VisVM
+steers VLMs away from generating sentences prone to hallucinations or
+insufficient detail, thereby producing higher quality responses. Experimental
+results demonstrate that VisVM-guided search significantly enhances VLMs'
+ability to generate descriptive captions with richer visual details and fewer
+hallucinations, compared with greedy decoding and search methods with other
+visual reward signals. Furthermore, we find that self-training the model with
+the VisVM-guided captions improve VLM's performance across a wide range of
+multimodal benchmarks, indicating the potential for developing self-improving
+VLMs. Our value model and code are available at
+https://github.com/si0wang/VisVM.
+
+
+
+
+ + ♻ ☆ Logic Agent: Enhancing Validity with Logic Rule Invocation +
+ +
+ Chain-of-Thought (CoT) prompting has emerged as a pivotal technique for
+augmenting the inferential capabilities of language models during reasoning
+tasks. Despite its advancements, CoT often grapples with challenges in
+validating reasoning validity and ensuring informativeness. Addressing these
+limitations, this paper introduces the Logic Agent (LA), an agent-based
+framework aimed at enhancing the validity of reasoning processes in Large
+Language Models (LLMs) through strategic logic rule invocation. Unlike
+conventional approaches, LA transforms LLMs into logic agents that dynamically
+apply propositional logic rules, initiating the reasoning process by converting
+natural language inputs into structured logic forms. The logic agent leverages
+a comprehensive set of predefined functions to systematically navigate the
+reasoning process. This methodology not only promotes the structured and
+coherent generation of reasoning constructs but also significantly improves
+their interpretability and logical coherence. Through extensive
+experimentation, we demonstrate LA's capacity to scale effectively across
+various model sizes, markedly improving the precision of complex reasoning
+across diverse tasks.
+
+
+
+ comment: The experiment is subject to certain errors
+
+
+
+ + ♻ ☆ Tulu 3: Pushing Frontiers in Open Language Model Post-Training +
+
+
+
+
+
+
+
+ Nathan Lambert, Jacob Morrison, Valentina Pyatkin, Shengyi Huang, Hamish Ivison, Faeze Brahman, Lester James V. Miranda, Alisa Liu, Nouha Dziri, Shane Lyu, Yuling Gu, Saumya Malik, Victoria Graf, Jena D. Hwang, Jiangjiang Yang, Ronan Le Bras, Oyvind Tafjord, Chris Wilhelm, Luca Soldaini, Noah A. Smith, Yizhong Wang, Pradeep Dasigi, Hannaneh Hajishirzi
+
+
+ Language model post-training is applied to refine behaviors and unlock new
+skills across a wide range of recent language models, but open recipes for
+applying these techniques lag behind proprietary ones. The underlying training
+data and recipes for post-training are simultaneously the most important pieces
+of the puzzle and the portion with the least transparency. To bridge this gap,
+we introduce Tulu 3, a family of fully-open state-of-the-art post-trained
+models, alongside its data, code, and training recipes, serving as a
+comprehensive guide for modern post-training techniques. Tulu 3, which builds
+on Llama 3.1 base models, achieves results surpassing the instruct versions of
+Llama 3.1, Qwen 2.5, Mistral, and even closed models such as GPT-4o-mini and
+Claude 3.5-Haiku. The training algorithms for our models include supervised
+finetuning (SFT), Direct Preference Optimization (DPO), and a novel method we
+call Reinforcement Learning with Verifiable Rewards (RLVR). With Tulu 3, we
+introduce a multi-task evaluation scheme for post-training recipes with
+development and unseen evaluations, standard benchmark implementations, and
+substantial decontamination of existing open datasets on said benchmarks. We
+conclude with analysis and discussion of training methods that did not reliably
+improve performance.
+ In addition to the Tulu 3 model weights and demo, we release the complete
+recipe -- including datasets for diverse core skills, a robust toolkit for data
+curation and evaluation, the training code and infrastructure, and, most
+importantly, a detailed report for reproducing and further adapting the Tulu 3
+approach to more domains.
+
+
+
+
+ + ♻ ☆ ELBA: Learning by Asking for Embodied Visual Navigation and Task + Completion WACV 2025 +
+ +
+ The research community has shown increasing interest in designing intelligent
+embodied agents that can assist humans in accomplishing tasks. Although there
+have been significant advancements in related vision-language benchmarks, most
+prior work has focused on building agents that follow instructions rather than
+endowing agents the ability to ask questions to actively resolve ambiguities
+arising naturally in embodied environments. To address this gap, we propose an
+Embodied Learning-By-Asking (ELBA) model that learns when and what questions to
+ask to dynamically acquire additional information for completing the task. We
+evaluate ELBA on the TEACh vision-dialog navigation and task completion
+dataset. Experimental results show that the proposed method achieves improved
+task performance compared to baseline models without question-answering
+capabilities.
+
+
+
+ comment: 14 pages, 10 figures, WACV 2025
+
+