AstroLLaMA-Chat: Scaling AstroLLaMA with Conversational and Diverse Datasets. (arXiv:2401.01916v1 [astro-ph.IM])
<a href="http://arxiv.org/find/astro-ph/1/au:+Perkowski_E/0/1/0/all/0/1">Ernest Perkowski</a>, <a href="http://arxiv.org/find/astro-ph/1/au:+Pan_R/0/1/0/all/0/1">Rui Pan</a>, <a href="http://arxiv.org/find/astro-ph/1/au:+Nguyen_T/0/1/0/all/0/1">Tuan Dung Nguyen</a>, <a href="http://arxiv.org/find/astro-ph/1/au:+Ting_Y/0/1/0/all/0/1">Yuan-Sen Ting</a>, <a href="http://arxiv.org/find/astro-ph/1/au:+Kruk_S/0/1/0/all/0/1">Sandor Kruk</a>, <a href="http://arxiv.org/find/astro-ph/1/au:+Zhang_T/0/1/0/all/0/1">Tong Zhang</a>, <a href="http://arxiv.org/find/astro-ph/1/au:+ONeill_C/0/1/0/all/0/1">Charlie O'Neill</a>, <a href="http://arxiv.org/find/astro-ph/1/au:+Jablonska_M/0/1/0/all/0/1">Maja Jablonska</a>, <a href="http://arxiv.org/find/astro-ph/1/au:+Smith_M/0/1/0/all/0/1">Michael J. Smith</a>, <a href="http://arxiv.org/find/astro-ph/1/au:+Schawinski_K/0/1/0/all/0/1">Kevin Schawinski</a>, <a href="http://arxiv.org/find/astro-ph/1/au:+Iyer_K/0/1/0/all/0/1">Kartheik Iyer</a>, <a href="http://arxiv.org/find/astro-ph/1/au:+UniverseTBD_I/0/1/0/all/0/1">Ioana Ciucă for UniverseTBD</a>
We explore the potential of enhancing LLM performance in astronomy-focused
question-answering through targeted, continual pre-training. By employing a
compact 7B-parameter LLaMA-2 model and focusing exclusively on a curated set of
astronomy corpus — comprising abstracts, introductions, and conclusions — we
achieve notable improvements in specialized topic comprehension. While general
LLMs like GPT-4 outperform in broader question-answering scenarios due to
superior reasoning capabilities, our findings suggest that continual
pre-training with limited resources can still enhance model performance on
specialized topics. Additionally, we present an extension of AstroLLaMA: the
fine-tuning of the 7B LLaMA model on a domain-specific conversational dataset,
culminating in the release of the chat-enabled AstroLLaMA for community use.
Comprehensive quantitative benchmarking is currently in progress and will be
detailed in an upcoming full paper. The model, AstroLLaMA-Chat, is now
available at https://huggingface.co/universeTBD, providing the first
open-source conversational AI tool tailored for the astronomy community.
We explore the potential of enhancing LLM performance in astronomy-focused
question-answering through targeted, continual pre-training. By employing a
compact 7B-parameter LLaMA-2 model and focusing exclusively on a curated set of
astronomy corpus — comprising abstracts, introductions, and conclusions — we
achieve notable improvements in specialized topic comprehension. While general
LLMs like GPT-4 outperform in broader question-answering scenarios due to
superior reasoning capabilities, our findings suggest that continual
pre-training with limited resources can still enhance model performance on
specialized topics. Additionally, we present an extension of AstroLLaMA: the
fine-tuning of the 7B LLaMA model on a domain-specific conversational dataset,
culminating in the release of the chat-enabled AstroLLaMA for community use.
Comprehensive quantitative benchmarking is currently in progress and will be
detailed in an upcoming full paper. The model, AstroLLaMA-Chat, is now
available at https://huggingface.co/universeTBD, providing the first
open-source conversational AI tool tailored for the astronomy community.
http://arxiv.org/icons/sfx.gif
