Evaluation and safety terms
How models are measured and how they get attacked: benchmarks, prompt injection, jailbreaking, and what alignment actually refers to.
llm benchmarks.
LLM benchmarks are standardised test sets and scoring harnesses used to compare model capabilities on tasks such as coding, reasoning, and factual knowledge. Leaderboards aggregate the scores; results guide model choice but degrade when test data leaks into training sets.
A benchmark is a fixed exam: a set of tasks, a scoring rule, and a number at the end. Public leaderboards such as LiveBench (livebench.ai) and Artificial Analysis (artificialanalysis.ai) aggregate scores across such exams, alongside price and speed, which is why they dominate the search results for this term.
The known failure mode is contamination: a benchmark that leaked into training data measures memory, not capability, which is why newer leaderboards rotate questions or use held-out sets. The subtler failure is relevance: a model that tops a maths benchmark can still be mediocre inside your editor, because agentic coding depends on tool use, context handling, and recovery from its own errors, none of which a one-shot exam measures.
How to use benchmarks as a builder: treat them as a shortlist generator, never as a verdict. The model that wins your workload is the one that wins on your own tasks, which is a small private eval you run yourself; leaderboard position tells you which two or three models are worth the afternoon it takes to test them against the tools you already use.
prompt injection.
Prompt injection is an attack where untrusted content an AI system reads, such as a web page, email, or document, contains instructions the model follows as if they came from the user, potentially exfiltrating data or triggering unintended tool calls.
The attack works because a language model receives instructions and data in the same channel: tokens. Anything the model reads, a scraped web page, a pasted document, a tool result, can contain text shaped like instructions, and the model has no reliable way to know the author was not you. OWASP ranks prompt injection first in its Top 10 for LLM applications (genai.owasp.org).
The risk scales with what the model can do. A chatbot that only talks can be embarrassed; an agent with tool access can be steered. The dangerous pattern is the combination security researchers keep flagging: access to private data, exposure to untrusted content, and the ability to send data out. An agent with all three can be tricked into exfiltrating what it was trusted with.
Defence for vibe coders is architectural, not a clever system prompt: treat all fetched content as untrusted, keep tool permissions minimal, require confirmation for irreversible actions, and assume the injection will sometimes work when deciding what the agent may reach. Our agent security scanning guide covers the tooling layer on top of those basics.
llm jailbreaking.
Jailbreaking is crafting inputs that bypass a model's safety training so it produces responses its provider intended it to refuse. Providers patch known techniques continuously, and defenders study jailbreaks to harden systems that expose models to untrusted users.
Where prompt injection hides instructions in content the model reads, jailbreaking is the user attacking the model directly, with role-play framing, encoding tricks, or many-step conversations that walk the model away from its refusals. OWASP's LLM Top 10 lists it alongside injection as a first-class application risk (genai.owasp.org).
Why a builder should care even without malicious users: any app that exposes a model to the public inherits this attack surface. If your support bot can be walked into off-brand or harmful output, the screenshot carries your logo, not the model vendor's. Providers continuously patch known techniques at the model layer, but the application layer, what your product does with the output, is yours.
The defensive posture is the same discipline as injection: constrain what the model can say and do in your product, filter or review outputs where the stakes justify it, and log conversations so you learn about the failure before your users' feeds do. Treat published jailbreaks as free red-team reports about behaviours your own guardrails should catch.
model alignment.
Alignment is the effort to make AI systems pursue their operators' intended goals and human values, using techniques such as reinforcement learning from human feedback. In day-to-day product terms it covers helpfulness, honesty, and refusing harmful requests.
The canonical technique behind the term as most builders meet it is reinforcement learning from human feedback, described in the InstructGPT paper that turned raw next-token predictors into instruction-following assistants (arXiv:2203.02155). Human raters score outputs, a reward model learns those preferences, and the base model is trained against it.
For a builder the practical surface of alignment is model behaviour you did not write: the assistant refuses some requests, hedges others, and follows system prompts with a deference that varies by vendor and version. When a model update changes how compliant or cautious your product feels, that is the alignment layer moving underneath you, which is a concrete reason to pin model versions and run a regression eval before upgrading.
The word also names a research field concerned with harder, longer-horizon questions about controlling increasingly capable systems. That debate matters, but do not let the grand version obscure the everyday one: for your product, alignment is whether the model's trained behaviour and your instructions produce what your users need, and it is testable like anything else.