Publications | Kanghee Park

Constrained Sampling for Language Models Should Be Easy: An MCMC Perspective

Mon, 01 Dec 2025 00:00:00 +0000

This paper studies constrained sampling for language models from an MCMC perspective. The goal is to generate outputs that satisfy hard constraints while preserving the language model’s conditional distribution, instead of only forcing validity through local token masks.

The work frames constrained generation as a sampling problem over valid outputs. It constructs proposal distributions that stay inside the constrained space and uses a Metropolis-Hastings acceptance rule based on the language model likelihood. This gives a principled way to explore valid outputs while converging toward the desired conditional distribution.

The paper evaluates the approach on synthetic benchmarks and program fuzzing tasks, where both validity and diversity matter. It appeared at NeurIPS 2025.

LOUD: Synthesizing Strongest and Weakest Specifications

Wed, 01 Oct 2025 00:00:00 +0000

This work extends the specification-synthesis framework from my to a more general setting. The earlier framework synthesized strongest properties for quantifier-free queries; LOUD handles queries with existential quantifiers and supports both over-approximating and under-approximating specifications.

This matters for programs with nondeterministic or concurrent behavior, where useful specifications can describe what must happen across all executions as well as what can happen along some execution. LOUD captures these two views through strongest L-consequences and weakest L-implicants.

The paper introduces the LOUD framework and the ASPIRE solver. The evaluation uses the framework to describe and localize bugs, extract properties from concurrent programs, and synthesize winning strategies in two-player games.

Flexible and Efficient Grammar-Constrained Decoding

Tue, 01 Jul 2025 00:00:00 +0000

This paper focuses on the practical cost of grammar-constrained decoding. Grammar-constrained decoding can guarantee that generated outputs follow a context-free grammar, but efficient implementations must reconcile two different token systems: the grammar’s terminals and the language model’s subword tokenizer.

The work introduces a more flexible and efficient algorithm for computing the token masks needed during decoding. The main improvement is in offline preprocessing, making it faster to prepare common grammars while preserving efficient online mask computation during generation.

This is useful for structured generation tasks such as code, data formats, and other domains where syntax errors are unacceptable. The work appeared at ICML 2025.

Grammar-Aligned Decoding

Sun, 01 Dec 2024 00:00:00 +0000

Some existing works constrain possible next tokens to ensure grammaticality of generated output. But validity alone is not enough: a decoding algorithm should also preserve the language model’s distribution after conditioning on the constraint.

For example, suppose I want the output to always be one of the following numbers: {00000, 10000, 10001, 10010, 10011, 10100, 10101, 10111, 11001, 11010, 11011, 11100, 11101, 11111} If the original LLM gives each possible length-5 binary string equal probability, the desired constrained distribution should be uniform over the valid strings above. A locally constrained decoder, however, may choose 0 or 1 with equal probability at the first step and then force completion after choosing 0, causing 00000 to appear far too often.

Grammar-aligned decoding formalizes this mismatch. Given a constraint G, we want to sample according to the original LLM probability conditioned on satisfying G, not according to a distribution induced by greedy token masking.

The paper proposes ASAp, a decoding algorithm that uses prior samples to approximate the future grammaticality of prefixes. ASAp guarantees grammatical outputs while aligning samples with the model’s conditional distribution, and the evaluation studies structured NLP and code-generation tasks.

Modular System Synthesis

Sun, 01 Oct 2023 00:00:00 +0000

This paper studies how to make program synthesis scale by exploiting modularity. Instead of synthesizing a large system in one monolithic search, the approach builds larger programs from smaller synthesized components.

The key idea is to structure the decomposition so that each subproblem remains similar in character to the original synthesis problem. This keeps the search spaces at different levels comparable and gives synthesis tools a path toward handling larger systems.

The work points toward synthesis workflows where modular system design is part of the synthesis process itself. It appeared at FMCAD 2023.

Synthesizing Specifications

Sun, 01 Oct 2023 00:00:00 +0000

This paper studies how to synthesize useful formal specifications from program code. Instead of requiring a developer to manually write every property, the framework takes a query about a set of function definitions and a domain-specific language for expressing candidate properties.

The synthesized results are best properties in the chosen language: each property is as precise as possible within the language, and the set is exhaustive with respect to the query. This makes the approach useful for extracting program facts that are both formal and tailored to the kind of property a user wants to see.

The work is implemented in Spyro and demonstrates applications including program-specification mining, abstract-domain operations, and algebraic properties of program modules. It appeared at OOPSLA 2023.