{"id":18163,"date":"2026-06-29T04:03:24","date_gmt":"2026-06-29T04:03:24","guid":{"rendered":"https:\/\/megagon.ai\/?post_type=publications&p=18163"},"modified":"2026-06-29T04:17:23","modified_gmt":"2026-06-29T04:17:23","slug":"probing-the-dynamics-of-syntactic-ability-acquisition-throughout-llm-pretraining","status":"publish","type":"publications","link":"https:\/\/megagon.ai\/jp\/publications\/probing-the-dynamics-of-syntactic-ability-acquisition-throughout-llm-pretraining\/","title":{"rendered":"Probing the Dynamics of Syntactic Ability Acquisition Throughout LLM Pretraining"},"template":"","publications-tags":[326],"conference-year":[353],"conference":[394],"class_list":["post-18163","publications","type-publications","status-publish","hentry","publications-tags-llm-nlp-jp","conference-year-353","conference-udw"],"yoast_head":"\nProbing the Dynamics of Syntactic Ability Acquisition Throughout LLM Pretraining - Megagon<\/title>\n<meta name=\"description\" content=\"In this research, we introduce LoRA probing, a lightweight approach for observing how core syntactic abilities emergeduring LLM pretraining. Leveraging OLMo-2\u2019s public intermediate checkpoints, we trace learning curves across 24 pretraining stages on 33 Universal Dependencies languages by fine-tuning LoRA with step-by-step parsinginstructions and a simple tabular output. To fit the relatively short context length of the OLMo-2, we design a compact 2-step-no-form prompt template and this matches the baseline in average accuracy while halving the context length and substantially increasing throughput, enabling efficient large-scale evaluation. Token Recall surpasses 0.9 within the first 1\u20132K pretraining steps, indicating that stable output formatting emerges early. Despite OLMo-2-7B\u2019s English-centric pretraining, LAS exceeds 80 points in 29 of 33 languages; however, relations such as iobj and csubj show delayed onset and instability across many languages. LoRA probing thus provides a practical, reproducible lens on the cross-lingual dynamics of syntactic acquisition during LLM pretraining.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/megagon.ai\/jp\/publications\/probing-the-dynamics-of-syntactic-ability-acquisition-throughout-llm-pretraining\/\" \/>\n<meta property=\"og:locale\" content=\"ja_JP\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Tool-Induced Myopia (TIM) and the PyMath benchmark, accepted to ACL 2026 main conference\" \/>\n<meta property=\"og:description\" content=\"Tool-augmented LLMs can boost accuracy by calling external tools, but these gains don\u2019t usually reflect better reasoning. Our study shows that while tool use improves final-answer accuracy by up to 19.3 points, it comes at the cost of degraded reasoning quality.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/megagon.ai\/jp\/publications\/probing-the-dynamics-of-syntactic-ability-acquisition-throughout-llm-pretraining\/\" \/>\n<meta property=\"og:site_name\" content=\"Megagon\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/megagonlabs\/\" \/>\n<meta property=\"article:modified_time\" content=\"2026-06-29T04:17:23+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/megagon.ai\/wp-content\/uploads\/2025\/02\/Publications.png\" \/>\n\t<meta property=\"og:image:width\" content=\"1600\" \/>\n\t<meta property=\"og:image:height\" content=\"900\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/png\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:title\" content=\"Tool-Induced Myopia (TIM) and the PyMath benchmark: paper accepted to ACL 2026 main conference\" \/>\n<meta name=\"twitter:description\" content=\"Tool-augmented LLMs can boost accuracy by calling external tools, but these gains don\u2019t usually reflect better reasoning. Our study shows that while tool use improves final-answer accuracy by up to 19.3 points, it comes at the cost of degraded reasoning quality.\" \/>\n<meta name=\"twitter:image\" content=\"https:\/\/megagon.ai\/wp-content\/uploads\/2025\/02\/Publications.png\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/megagon.ai\\\/jp\\\/publications\\\/probing-the-dynamics-of-syntactic-ability-acquisition-throughout-llm-pretraining\\\/\",\"url\":\"https:\\\/\\\/megagon.ai\\\/jp\\\/publications\\\/probing-the-dynamics-of-syntactic-ability-acquisition-throughout-llm-pretraining\\\/\",\"name\":\"Probing the Dynamics of Syntactic Ability Acquisition Throughout LLM Pretraining - Megagon\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/megagon.ai\\\/jp\\\/#website\"},\"datePublished\":\"2026-06-29T04:03:24+00:00\",\"dateModified\":\"2026-06-29T04:17:23+00:00\",\"description\":\"In this research, we introduce LoRA probing, a lightweight approach for observing how core syntactic abilities emergeduring LLM pretraining. Leveraging OLMo-2\u2019s public intermediate checkpoints, we trace learning curves across 24 pretraining stages on 33 Universal Dependencies languages by fine-tuning LoRA with step-by-step parsinginstructions and a simple tabular output. To fit the relatively short context length of the OLMo-2, we design a compact 2-step-no-form prompt template and this matches the baseline in average accuracy while halving the context length and substantially increasing throughput, enabling efficient large-scale evaluation. Token Recall surpasses 0.9 within the first 1\u20132K pretraining steps, indicating that stable output formatting emerges early. Despite OLMo-2-7B\u2019s English-centric pretraining, LAS exceeds 80 points in 29 of 33 languages; however, relations such as iobj and csubj show delayed onset and instability across many languages. LoRA probing thus provides a practical, reproducible lens on the cross-lingual dynamics of syntactic acquisition during LLM pretraining.\",\"breadcrumb\":{\"@id\":\"https:\\\/\\\/megagon.ai\\\/jp\\\/publications\\\/probing-the-dynamics-of-syntactic-ability-acquisition-throughout-llm-pretraining\\\/#breadcrumb\"},\"inLanguage\":\"ja-JP\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/megagon.ai\\\/jp\\\/publications\\\/probing-the-dynamics-of-syntactic-ability-acquisition-throughout-llm-pretraining\\\/\"]}]},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/megagon.ai\\\/jp\\\/publications\\\/probing-the-dynamics-of-syntactic-ability-acquisition-throughout-llm-pretraining\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home\",\"item\":\"https:\\\/\\\/megagon.ai\\\/jp\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Publications\",\"item\":\"https:\\\/\\\/megagon.ai\\\/jp\\\/publications\\\/\"},{\"@type\":\"ListItem\",\"position\":3,\"name\":\"Probing the Dynamics of Syntactic Ability Acquisition Throughout LLM Pretraining\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/megagon.ai\\\/jp\\\/#website\",\"url\":\"https:\\\/\\\/megagon.ai\\\/jp\\\/\",\"name\":\"Megagon Labs\",\"description\":\"\",\"publisher\":{\"@id\":\"https:\\\/\\\/megagon.ai\\\/jp\\\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/megagon.ai\\\/jp\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"ja-JP\"},{\"@type\":\"Organization\",\"@id\":\"https:\\\/\\\/megagon.ai\\\/jp\\\/#organization\",\"name\":\"Megagon Labs\",\"url\":\"https:\\\/\\\/megagon.ai\\\/\",\"logo\":{\"@type\":\"ImageObject\",\"url\":\"https:\\\/\\\/megagon.ai\\\/wp-content\\\/uploads\\\/2025\\\/02\\\/Logo-Megagon-Labs.webp\",\"caption\":\"Megagon Labs\"},\"image\":{\"url\":\"https:\\\/\\\/megagon.ai\\\/wp-content\\\/uploads\\\/2025\\\/02\\\/Logo-Megagon-Labs.webp\"},\"description\":\"Megagon Labs is an AI research organization conducting research in compound AI systems, large language models, data-AI symbiosis, and human-centered AI. Megagon Labs shares its findings with the broader community through open-source tools, datasets, publications, workshops, and an invited speaker series.\",\"sameAs\":[\"https:\\\/\\\/github.com\\\/megagonlabs\",\"https:\\\/\\\/twitter.com\\\/megagonlabs\",\"https:\\\/\\\/www.linkedin.com\\\/company\\\/megagon-labs\\\/\",\"https:\\\/\\\/www.facebook.com\\\/megagonlabs\\\/\"],\"address\":{\"@type\":\"PostalAddress\",\"streetAddress\":\"444 Castro Street\",\"addressLocality\":\"Mountain View\",\"addressRegion\":\"CA\",\"postalCode\":\"94041\",\"addressCountry\":\"US\"},\"contactPoint\":{\"@type\":\"ContactPoint\",\"email\":\"contactus@megagon.ai\",\"contactType\":\"general inquiries\"},\"parentOrganization\":{\"@type\":\"Organization\",\"name\":\"Recruit Holdings\",\"url\":\"https:\\\/\\\/recruit-holdings.com\\\/en\\\/\"}}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"Probing the Dynamics of Syntactic Ability Acquisition Throughout LLM Pretraining - Megagon","description":"In this research, we introduce LoRA probing, a lightweight approach for observing how core syntactic abilities emergeduring LLM pretraining. Leveraging OLMo-2\u2019s public intermediate checkpoints, we trace learning curves across 24 pretraining stages on 33 Universal Dependencies languages by fine-tuning LoRA with step-by-step parsinginstructions and a simple tabular output. To fit the relatively short context length of the OLMo-2, we design a compact 2-step-no-form prompt template and this matches the baseline in average accuracy while halving the context length and substantially increasing throughput, enabling efficient large-scale evaluation. Token Recall surpasses 0.9 within the first 1\u20132K pretraining steps, indicating that stable output formatting emerges early. Despite OLMo-2-7B\u2019s English-centric pretraining, LAS exceeds 80 points in 29 of 33 languages; however, relations such as iobj and csubj show delayed onset and instability across many languages. LoRA probing thus provides a practical, reproducible lens on the cross-lingual dynamics of syntactic acquisition during LLM pretraining.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/megagon.ai\/jp\/publications\/probing-the-dynamics-of-syntactic-ability-acquisition-throughout-llm-pretraining\/","og_locale":"ja_JP","og_type":"article","og_title":"Tool-Induced Myopia (TIM) and the PyMath benchmark, accepted to ACL 2026 main conference","og_description":"Tool-augmented LLMs can boost accuracy by calling external tools, but these gains don\u2019t usually reflect better reasoning. Our study shows that while tool use improves final-answer accuracy by up to 19.3 points, it comes at the cost of degraded reasoning quality.","og_url":"https:\/\/megagon.ai\/jp\/publications\/probing-the-dynamics-of-syntactic-ability-acquisition-throughout-llm-pretraining\/","og_site_name":"Megagon","article_publisher":"https:\/\/www.facebook.com\/megagonlabs\/","article_modified_time":"2026-06-29T04:17:23+00:00","og_image":[{"width":1600,"height":900,"url":"https:\/\/megagon.ai\/wp-content\/uploads\/2025\/02\/Publications.png","type":"image\/png"}],"twitter_card":"summary_large_image","twitter_title":"Tool-Induced Myopia (TIM) and the PyMath benchmark: paper accepted to ACL 2026 main conference","twitter_description":"Tool-augmented LLMs can boost accuracy by calling external tools, but these gains don\u2019t usually reflect better reasoning. Our study shows that while tool use improves final-answer accuracy by up to 19.3 points, it comes at the cost of degraded reasoning quality.","twitter_image":"https:\/\/megagon.ai\/wp-content\/uploads\/2025\/02\/Publications.png","schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"WebPage","@id":"https:\/\/megagon.ai\/jp\/publications\/probing-the-dynamics-of-syntactic-ability-acquisition-throughout-llm-pretraining\/","url":"https:\/\/megagon.ai\/jp\/publications\/probing-the-dynamics-of-syntactic-ability-acquisition-throughout-llm-pretraining\/","name":"Probing the Dynamics of Syntactic Ability Acquisition Throughout LLM Pretraining - Megagon","isPartOf":{"@id":"https:\/\/megagon.ai\/jp\/#website"},"datePublished":"2026-06-29T04:03:24+00:00","dateModified":"2026-06-29T04:17:23+00:00","description":"In this research, we introduce LoRA probing, a lightweight approach for observing how core syntactic abilities emergeduring LLM pretraining. Leveraging OLMo-2\u2019s public intermediate checkpoints, we trace learning curves across 24 pretraining stages on 33 Universal Dependencies languages by fine-tuning LoRA with step-by-step parsinginstructions and a simple tabular output. To fit the relatively short context length of the OLMo-2, we design a compact 2-step-no-form prompt template and this matches the baseline in average accuracy while halving the context length and substantially increasing throughput, enabling efficient large-scale evaluation. Token Recall surpasses 0.9 within the first 1\u20132K pretraining steps, indicating that stable output formatting emerges early. Despite OLMo-2-7B\u2019s English-centric pretraining, LAS exceeds 80 points in 29 of 33 languages; however, relations such as iobj and csubj show delayed onset and instability across many languages. LoRA probing thus provides a practical, reproducible lens on the cross-lingual dynamics of syntactic acquisition during LLM pretraining.","breadcrumb":{"@id":"https:\/\/megagon.ai\/jp\/publications\/probing-the-dynamics-of-syntactic-ability-acquisition-throughout-llm-pretraining\/#breadcrumb"},"inLanguage":"ja-JP","potentialAction":[{"@type":"ReadAction","target":["https:\/\/megagon.ai\/jp\/publications\/probing-the-dynamics-of-syntactic-ability-acquisition-throughout-llm-pretraining\/"]}]},{"@type":"BreadcrumbList","@id":"https:\/\/megagon.ai\/jp\/publications\/probing-the-dynamics-of-syntactic-ability-acquisition-throughout-llm-pretraining\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home","item":"https:\/\/megagon.ai\/jp\/"},{"@type":"ListItem","position":2,"name":"Publications","item":"https:\/\/megagon.ai\/jp\/publications\/"},{"@type":"ListItem","position":3,"name":"Probing the Dynamics of Syntactic Ability Acquisition Throughout LLM Pretraining"}]},{"@type":"WebSite","@id":"https:\/\/megagon.ai\/jp\/#website","url":"https:\/\/megagon.ai\/jp\/","name":"Megagon Labs","description":"","publisher":{"@id":"https:\/\/megagon.ai\/jp\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/megagon.ai\/jp\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"ja-JP"},{"@type":"Organization","@id":"https:\/\/megagon.ai\/jp\/#organization","name":"Megagon Labs","url":"https:\/\/megagon.ai\/","logo":{"@type":"ImageObject","url":"https:\/\/megagon.ai\/wp-content\/uploads\/2025\/02\/Logo-Megagon-Labs.webp","caption":"Megagon Labs"},"image":{"url":"https:\/\/megagon.ai\/wp-content\/uploads\/2025\/02\/Logo-Megagon-Labs.webp"},"description":"Megagon Labs is an AI research organization conducting research in compound AI systems, large language models, data-AI symbiosis, and human-centered AI. Megagon Labs shares its findings with the broader community through open-source tools, datasets, publications, workshops, and an invited speaker series.","sameAs":["https:\/\/github.com\/megagonlabs","https:\/\/twitter.com\/megagonlabs","https:\/\/www.linkedin.com\/company\/megagon-labs\/","https:\/\/www.facebook.com\/megagonlabs\/"],"address":{"@type":"PostalAddress","streetAddress":"444 Castro Street","addressLocality":"Mountain View","addressRegion":"CA","postalCode":"94041","addressCountry":"US"},"contactPoint":{"@type":"ContactPoint","email":"contactus@megagon.ai","contactType":"general inquiries"},"parentOrganization":{"@type":"Organization","name":"Recruit Holdings","url":"https:\/\/recruit-holdings.com\/en\/"}}]}},"_links":{"self":[{"href":"https:\/\/megagon.ai\/jp\/wp-json\/wp\/v2\/publications\/18163","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/megagon.ai\/jp\/wp-json\/wp\/v2\/publications"}],"about":[{"href":"https:\/\/megagon.ai\/jp\/wp-json\/wp\/v2\/types\/publications"}],"version-history":[{"count":1,"href":"https:\/\/megagon.ai\/jp\/wp-json\/wp\/v2\/publications\/18163\/revisions"}],"predecessor-version":[{"id":18164,"href":"https:\/\/megagon.ai\/jp\/wp-json\/wp\/v2\/publications\/18163\/revisions\/18164"}],"wp:attachment":[{"href":"https:\/\/megagon.ai\/jp\/wp-json\/wp\/v2\/media?parent=18163"}],"wp:term":[{"taxonomy":"publications-tags","embeddable":true,"href":"https:\/\/megagon.ai\/jp\/wp-json\/wp\/v2\/publications-tags?post=18163"},{"taxonomy":"conference-year","embeddable":true,"href":"https:\/\/megagon.ai\/jp\/wp-json\/wp\/v2\/conference-year?post=18163"},{"taxonomy":"conference","embeddable":true,"href":"https:\/\/megagon.ai\/jp\/wp-json\/wp\/v2\/conference?post=18163"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}