Rubrics are structured evaluation criteria used to assess the quality of model outputs across defined dimensions such as accuracy, relevance, coherence, and factual grounding. Rubrics provide a systematic framework for human annotation, automated scoring, and benchmarking of generative AI systems, including large language models. They are applied in tasks such as summarization evaluation, open-ended question answering, and the development of reward models for reinforcement learning from human feedback.

Posts

RunAgent: Interpreting Natural-Language Plans with Constraint-Guided Execution (IEEE)

/in /by

Humans solve problems by executing targeted plans, yet large language models (LLMs) remain unreliable for structured workflow execution. We propose RunAgent, a multiagent plan execution platform that interprets natural-language plans while enforcing stepwise execution through constraints and rubrics. RunAgent bridges the expressiveness of natural language and the determinism of programming languages via an agentic language with explicit control constructs (e.g., IF, GOTO, FORALL). It autonomously derives and verifies constraints at each step; dynamically selects among LLM reasoning, tool use, and Python execution; and integrates error correction to ensure correctness. Evaluations on Natural-plan and SciBench Datasets demonstrate that RunAgent outperforms baseline LLMs and state-of-the-art PlanGEN methods.

Read more