Constrained Decoding | Kanghee Park

Constrained Sampling for Language Models Should Be Easy: An MCMC Perspective

Mon, 01 Dec 2025 00:00:00 +0000

This paper studies constrained sampling for language models from an MCMC perspective. The goal is to generate outputs that satisfy hard constraints while preserving the language model’s conditional distribution, instead of only forcing validity through local token masks.

The work frames constrained generation as a sampling problem over valid outputs. It constructs proposal distributions that stay inside the constrained space and uses a Metropolis-Hastings acceptance rule based on the language model likelihood. This gives a principled way to explore valid outputs while converging toward the desired conditional distribution.

The paper evaluates the approach on synthetic benchmarks and program fuzzing tasks, where both validity and diversity matter. It appeared at NeurIPS 2025.

Flexible and Efficient Grammar-Constrained Decoding

Tue, 01 Jul 2025 00:00:00 +0000

This paper focuses on the practical cost of grammar-constrained decoding. Grammar-constrained decoding can guarantee that generated outputs follow a context-free grammar, but efficient implementations must reconcile two different token systems: the grammar’s terminals and the language model’s subword tokenizer.

The work introduces a more flexible and efficient algorithm for computing the token masks needed during decoding. The main improvement is in offline preprocessing, making it faster to prepare common grammars while preserving efficient online mask computation during generation.

This is useful for structured generation tasks such as code, data formats, and other domains where syntax errors are unacceptable. The work appeared at ICML 2025.

Grammar-Aligned Decoding

Sun, 01 Dec 2024 00:00:00 +0000

Some existing works constrain possible next tokens to ensure grammaticality of generated output. But validity alone is not enough: a decoding algorithm should also preserve the language model’s distribution after conditioning on the constraint.

For example, suppose I want the output to always be one of the following numbers: {00000, 10000, 10001, 10010, 10011, 10100, 10101, 10111, 11001, 11010, 11011, 11100, 11101, 11111} If the original LLM gives each possible length-5 binary string equal probability, the desired constrained distribution should be uniform over the valid strings above. A locally constrained decoder, however, may choose 0 or 1 with equal probability at the first step and then force completion after choosing 0, causing 00000 to appear far too often.

Grammar-aligned decoding formalizes this mismatch. Given a constraint G, we want to sample according to the original LLM probability conditioned on satisfying G, not according to a distribution induced by greedy token masking.

The paper proposes ASAp, a decoding algorithm that uses prior samples to approximate the future grammaticality of prefixes. ASAp guarantees grammatical outputs while aligning samples with the model’s conditional distribution, and the evaluation studies structured NLP and code-generation tasks.