Assessing Large Language Models: Architectural Archive Metadata and Transcription

Our collections often remain frustratingly separate from one another. The promise of increased access to GLAMS (Galleries, Libraries, Archives, Museums, Special Collections), including efforts in digital scholarship and linked data, can only extend to those items that are thoroughly and accurately described. This work is labor and time intensive, and many collections are either minimally described or set aside in the hopes of securing more staff or funding. Advancements in artificial intelligence (AI) could be one of many tools to address this issue.

Exploration of LLMs in metadata and cataloging workflows is ongoing. Recent OCR studies show promise: Greif et al. (2025) [1] found Gemini 2.0 Flash and GPT-4o matched traditional OCR tools like Transkribus and Tesseract, while Thomas et al. (2024) [2] showed Llama 2 outperformed a fine-tuned BART in post-OCR error correction. Chen and Li’s (2024) [3] survey of cataloging professionals revealed concerns about reliability and potential for de-professionalization of the field. The Library of Congress recommends members evaluate AI projects by questioning if implementation would be “responsible, effective, and practical” (Brador, 2024) [4]. Several studies emphasize the importance of human oversight. Huang et al. (2024) [5] found GPT-4o was cost-effective and scalable for web archive metadata generation but produced more inaccuracies and hallucinations than human-curated metadata. Toth et al. (2025) [6] found ChatGPT-4 valuable for revealing hidden structures in archival data despite occasional hallucinations that required human verification. Building on this work, we utilized core principles of the PROMPT design framework by creating a persona for the LLMs and contextualizing requests through detailed requirements, format, intended purpose, audience, and professional expectations such as tone (Hartman-Caverly, 2024) [7].

Our research assesses the performance of seven LLMs in the generation of metadata and transcriptions for archival materials. We tested 14 images of historical architecture magazine pages and previously created optical character recognition (OCR) text of those pages. Our study tested the LLMs via Application Programming Interface (API) in an automated workflow as well as a web-based interface. In this paper, we describe our process of evaluation, analysis of the efficacy of LLMs in various description tasks, recommendations for implementation, and ethical and practical considerations of AI metadata projects.

Our cross-departmental team included six staff members and student workers with expertise in metadata, digital collections, AI implementation, archives, architectural history, and digital scholarship. This approach allowed us to develop comprehensive evaluation criteria and assessment methods. The project focused on the Southern Architect and Building News (SABN) collection (1882-1932), a monthly trade publication documenting architecture and building trades in the southern United States. It is comprised of 254 distinct issues that have been digitized with basic metadata but not fully indexed or described. More robust metadata will address recent research interest expressed at our university. The issues chosen contain illustrated covers, advertisements, and editorial content, and incorporate a diverse range of fonts, symbols, photographs, and drawings.

Through group discussion and review of local metadata best practices, we established three primary evaluation criteria: completeness, or whether descriptions fully described the contents of a page; accuracy, the correctness of the OCR text, descriptions, terms and entities when compared to the original item; and usefulness, or the increased value for search. Similar criteria have been employed in Zavalin and Zavalina (2025) [8] and Shyr et al (2024) [9]. Based off of these three primary criteria, we scored results using a scale of 1-4. Blank responses returned by models were given a 0.

We tested models’ abilities to perform five key tasks: OCR text editing and transcription; table of contents entry generation; named entity extraction; and Library of Congress Subject Heading (LCSH) generation. The web and API testing utilized different page selections strategically in order to evaluate diverse content types. Results were manually compiled into Excel spreadsheets for evaluation. We conducted blind evaluations wherein model names were obscured in testing spreadsheets to reduce tester bias for or against any models or companies. Each team member independently scored all outputs, and we maintained notes to capture qualitative observations.

Five models were tested through their official web interfaces: GPT-4o-mini, Google Gemini, Claude Sonnet, Haiku when Sonnet was briefly unavailable, Llama, and Microsoft Copilot (for the image input workflow only). Testing was conducted using free accounts, with the exception of Copilot, which was tested using an institutional account. We selected four pages from three different SABN issues: two full page editorials published in 1930, and two dual-column pages with multiple blocks of text published in 1928. Web interfaces were tested in two phases: text input testing (October 2024) and image input testing (February 2025). The API prompts were divided into sections due to limitations on unpaid accounts.

We tested six models via API: Claude Haiku, Claude 3 Sonnet, Claude Sonnet 3.5, GPT-4o-mini, GPT-4o, and Gemini 2.0 Flash Experimental (for the image input workflow only). We selected ten pages from a 1924 and a 1931 issue. The pages included a cover page; a contents page with advertising schedule; advertisements with visual elements and complex layouts; editorial content with multiple blocks of text; a page with a captioned photograph of a home in Alabama; and a text-heavy article. API testing was conducted in two phases: OCR text input testing (December 9, 2024) and image input testing (December 11-12, 2024). We developed Python scripts to automatically send input files with prompts to each LLM. The scripts captured the LLM outputs and wrote them to Excel spreadsheets for evaluation.

Sentiment analysis was performed on our scoring notes, 56 total, to evaluate trends and biases. Each project member’s annotations were compiled into a text document and imported into R Studio for analysis, using the sentimentr package. While testers scored the LLMs differently on an absolute scale, with some perhaps more likely to give high or low scores, the relative sentiment expressed in comments was consistent across testers for the various models.

Claude Sonnet 3.5 emerged as the top performer, achieving the highest overall scores (see Table 2 below.) The performance hierarchy was fairly consistent for each model regardless of testing method, but models performed better via API than web interface due to their relative stability. At the time of testing, Copilot only accepted PDFs, and Haiku was only available to unpaid accounts for a short time, resulting in some N/A values. Gemini and Llama returned few to no responses, with Gemini having difficulty in longer sessions and Llama not allowing file uploads. These are represented below with a 0.00 score.

In terms of specific tasks, OCR editing/transcription and content summarization were the greatest strengths of the LLMs (see Table 3). AI-generated OCR results outperformed our original OCR, in line with findings from Thomas et al (2024) [2]. Entity extraction was ranked as the second-lowest performing task, but testers felt it provided valuable metadata enhancement that could improve discoverability. Subject heading creation emerged as the weakest task across all models tested. The difficulty primarily stemmed from the complicated and variable syntax of generating exact matches of LCSH. Controlled vocabularies remain a challenge for LLMs, as found in Chow et al (2024) [10]. One surprise was that, though in general larger models produced the best results, Haiku performed best in content summarization.

Some content presented more challenges. Cover pages received low scores in entity extraction and table of contents creation. This was likely due to the limited textual content available for analysis. Transcribing and editing OCR for text-heavy pages tended to have more small errors, though Claude 3.5 was an exception. Multi-column layouts and pages with mixed text and images could be a challenge for current LLMs, but they also showed strength when compared to traditional OCR technologies.

Models responded differently to a racist slur that was used in an advertisement published in the journal. In image-based API testing, Gemini and GPT-4o-mini refused to process the page.

In text-based API testing, all models processed the page but Claude Sonnet 3.5 substituted the offensive term with a neutral word during OCR cleaning. Complete refusal to process materials containing offensive historical language or uncritical substitution of terms could result in collection gaps (Antracoli et al., 2019) [11], rendering important historical materials undiscoverable. Alternatively, Claude 3 Sonnet and Haiku wrote the term in the content warning and short summary. Repeating harmful terms can result in psychological damage known as vicarious trauma (Wright & Laurent, 2021) [12]. GPT-4o was the only model that transcribed the term across API workflows without repeating it. Collections and communities vary, and LLM use will need to be fitted to specific circumstances (Antracoli et al., 2019; Wilson Special Collections Library, 2022) [11], [13]. Importantly, web-based models carry the risk that input data will be used for training, risking exposure and replication of sensitive information.

In general, larger models produced better output in our testing. Claude Sonnet 3.5 delivered the highest quality scores but at a higher cost, with image processing at $14.16 and text at $11.18 per 1,000 pages. GPT-4o-mini offered the best cost efficiency, with image at $1.15 and text at $0.52 per 1,000 pages, though it had lower overall quality. Notably, Haiku performed especially well in summarization despite being a smaller and more affordable model at $0.86 for image and $1.34 for text per 1,000 pages.

GPT-4o had the longest processing times regardless of format, followed by GPT-4o-mini for image processing. Running 1,000 pages with GPT-4o would take about 5.5 hours, while Haiku could handle the same volume in under two hours. Image processing was generally slower than text, though the difference was sometimes negligible. Gemini and Haiku were consistently the fastest models. For large-scale projects, batch processing can reduce costs by up to 50 percent while ensuring results within 24 hours.

Text inputs are generally more cost-effective than images, but overall cost difference may vary depending upon collection size and model. For example, Claude 3 Sonnet costs about $6.27 more per 1,000 pages for image analysis, while Haiku costs $0.48 more. Anthropic models, including Claude Sonnet 3.5 and Haiku, struggled with smaller text in images, likely due to lower-resolution image ingest during pre-processing compared to OpenAI and Gemini models.

Budget-conscious projects can use smaller LLMs through APIs or choose to use web-based chatbots, though reliability, quality, and data privacy require consideration. We recommend choosing models based on demonstrated strengths in specific tasks, collection requirements, and institutional access. Batch processing can be used via API to reduce costs, and results can be improved through prompt optimization, rigorous testing, and formalized human oversight protocols. Pre-processing and human evaluation should strive to match the growing best practice of including community stakeholders in metadata creation (University of California Irvine Libraries, 2025; Trans Metadata Collective et al., 2022) [14], [15]. Community awareness of ethical pitfalls may be more acute than that of archivists, often a result of lived experience as well as subject expertise.

Our research demonstrates that LLMs can offer promising capabilities for enhancing metadata creation in archival collections, though with limitations that require strategic implementation. A significant challenge will be hosting and supporting search of the increased volume of the metadata generated, including fields describing AI usage and mechanisms allowing users to flag potential inaccuracies. Other challenges include keeping up with the AI landscape, as noted in Cushing and Osti (2022) [16], navigating the stability and security of the tools available, and processing harmful content. This could prove to be very challenging for smaller or under-resourced institutions. We conclude that LLM workflows designed carefully and in accordance with archival best practices can increase access, enrich metadata, and allow libraries and archives to address processing backlogs.

We employed Anthropic, OpenAI, Gemini, and Llama models for the following purposes: generating metadata for our research and generating and debugging code used in the API workflow. We evaluated the metadata output through the methods outlined in this paper and thoroughly tested and debugged the code. The authors assume all responsibility for the content of this submission.

Denmark, E. R. (Ed.), & Harman, H. E. (Pres.). (1924, October). Southern Architect and Building News, 50(10). https://collections.lib.utexas.edu/catalog/utlarch:8fb93ee5-5b4b-4661-9a8c-ca0d76960182

Denmark, E. R. (Ed.), & Watts, B. H. (Ed.), & Harman, H. E., Jr. (Mgr.). (1928, February). Southern Architect and Building News, 54(2). https://collections.lib.utexas.edu/catalog/utlarch:db10bf40-401c-4cde-b845-0358fffcba54

Denmark, E. R. (Ed.), Harman, H. E., Jr. (Bus. Mgr.), & Sorrow, F. H. (Adv. Mgr.). (1930, March). Southern Architect and Building News, 56(3). https://collections.lib.utexas.edu/catalog/utlarch:69db3ebf-90b4-41b4-9967-5aca81bfecd3

Denmark, E. R. (Ed.), Harman, H. E., Jr. (Bus. Mgr.), & Sorrow, F. H. (Adv. Mgr.). (1930, August). Southern Architect and Building News, 56(8). https://collections.lib.utexas.edu/catalog/utlarch:6485cc23-e9e9-41e3-8ece-6052422ba1a6

Denmark, E. R. (Ed.), Harman, H. E., Jr. (Bus. Mgr.), & Sorrow, F. H. (Adv. Mgr.). (1931, December). Southern Architect and Building News, 57(12). https://collections.lib.utexas.edu/catalog/utlarch:b17ca698-bfaa-4855-ae0c-168d01ce5568