Trap Tasks: Why Standard Coding Challenges Fail with LLMs and How to Design Tests for Real Engineering Judgment

If you give a candidate a standard coding test today—reversing a linked list, building a RESTful todo API, or parsing a basic CSV file—something predictable happens:

The candidate pastes the prompt into Claude 3.7 or GPT-4o. The model generates a flawless implementation in four seconds. The candidate copies the code into the IDE, runs the test suite, watches all tests turn green, and submits.

Your evaluation process just gave a top score to a candidate who wrote zero lines of code, designed zero data structures, and spent zero seconds verifying edge cases.

Standard coding challenges are broken because they measure problem familiarity, an area where LLMs have near-infinite training data.

To evaluate authentic software engineering capability in the AI era, you must design Trap Tasks.

┌────────────────────────────────────────────────────────────────────────┐
│                        THE TRAP-TASK FORMULA                           │
│                                                                        │
│  [Naive One-Shot Prompt] ──────────► ❌ FAILS (Hidden Invariant Trap) │
│                                                                        │
│  [Steered + Verified Iteration] ───► ✅ PASSES (Objective Ground Truth)│
└────────────────────────────────────────────────────────────────────────┘

1. The Core Rule of AI Task Design

At Plaiback, we author and calibrate dozens of coding challenges for both Plaiback Hiring and Plaiback Kata. Every challenge in our catalog adheres to a foundational authoring rule:

The Verify-the-Trap Invariant: A technical challenge is only valid if a naive, unconstrained prompt to the default frontier model reliably fails the hidden test suite, while a carefully steered, human-verified solution passes.

If a frontier model can one-shot a challenge from a naive copy-paste of the prompt, the challenge is useless for evaluation. It cannot separate a senior engineer from someone who just clicks “Generate.”

2. The 3 Tiers of Trap Tasks

To test real engineering judgment across junior, mid-level, and senior levels, trap tasks are structured into three distinct tiers:

graph TD
    T1["Tier 1: Mechanical Traps (Warm-ups)"] --> T1E["Edge-case blindspots: Off-by-one backoff, URI encoding, JPY zero-decimals"]
    T2["Tier 2: Adversarial Debugging (Distrust)"] --> T2E["Broken starters, Green-but-wrong tests, Async race conditions"]
    T3["Tier 3: High-Altitude Under-Scoping (Steering)"] --> T3E["Cursor pagination drift, Algorithmic scaling O(n²), Prototype pollution"]

Tier 1: Mechanical Traps (The Edge-Case Blindspot)

Mechanical traps test whether a developer recognizes the standard failure modes of LLMs when generating common utility functions.

Example: retry-backoff

• The Prompt: “Implement an asynchronous retryWithBackoff(fn, options) utility with exponential backoff, jitter, and custom retry predicates.”
• The Default-Model Trap:
  1. Attempts vs. Retries: Models reliably confuse attempts: 3 (1 initial + 2 retries) with retries: 3 (1 initial + 3 retries).
  2. Trailing Delay: When the final attempt fails, naive model implementations still await sleep(delay) before re-throwing the error, causing unnecessary latency spikes.
  3. First vs. Last Error: In multi-attempt failures, naive code often caches and throws the first caught error rather than the most recent error that caused the final failure.

// The Trap in Action (Naive LLM output)
export async function retryWithBackoff(fn, { maxRetries, baseDelay }) {
  let lastError;
  for (let i = 0; i <= maxRetries; i++) {
    try {
      return await fn();
    } catch (err) {
      lastError = err;
      // ❌ TRAP: Executes delay even on the final iteration before throwing!
      await new Promise(res => setTimeout(res, baseDelay * Math.pow(2, i)));
    }
  }
  throw lastError;
}

A candidate who blindly trusts the green checkmark on basic tests will miss this trailing sleep. A candidate with verification discipline injects a mock clock to assert that total elapsed time on failure does not include a terminal sleep.

Tier 2: Adversarial Starter Traps (Distrust the Green Check)

Adversarial tasks test whether a candidate trusts the existing codebase blindly or actively verifies specifications against requirements.

Example: green-but-wrong

• The Setup: The candidate is given a legacy calculation module. They run npm test, and every single test passes with green checkmarks.
• The Trap: The existing test suite was written by a previous developer who misunderstood the business specification. The README specification states that date ranges are inclusive, but the existing test suite and code implement an exclusive range.
• The Test of Skill: If the candidate asks the agent to “Optimize the module while keeping tests passing”, the agent happily refactors the code to pass the incorrect tests.
• The Result: The candidate scores 0% on the hidden test suite. Only candidates who carefully read the README and update the test assertions to reflect the true business invariant succeed.

Example: concurrent-ledger

• The Challenge: Implement a high-throughput bank account balance ledger that handles concurrent deposit, withdrawal, and transfer requests.
• The Trap: Naive async JavaScript code performs const current = await db.getBalance(); await db.setBalance(current - amount);. Under concurrent Promise.all([withdraw(50), withdraw(50)]), this read-modify-write pattern causes lost updates and allows accounts to go negative.
• The Invariant: The candidate must steer the agent to build an in-memory sequential lock or serialized transaction queue without relying on third-party database locking libraries.

Tier 3: High-Altitude Under-Scoping (Architectural Steering)

Tier 3 challenges give the developer high-level requirements where the model’s natural instinct is to take architectural shortcuts.

Example: cursor-pagination

• The Challenge: Build an API endpoint that paginates a real-time activity feed where new events are continuously inserted at the top.
• The Trap: The model defaults to standard SQL OFFSET and LIMIT pagination. When new records arrive while a user is scrolling from page 1 to page 2, records shift downward, resulting in duplicate items and skipped records.
• The Invariant: The engineer must steer the agent toward an immutable opaque cursor (e.g., base64-encoded [timestamp, id]) that provides deterministic pagination regardless of real-time insertions.

// Naive Offset Pagination (Broken under real-time writes)
// Page 1: LIMIT 10 OFFSET 0  ──► Items [1..10]
// [New Item 0 inserted at top]
// Page 2: LIMIT 10 OFFSET 10 ──► Item 10 is returned AGAIN!

// Correct Steered Cursor Implementation
// Page 2: WHERE (created_at, id) < (cursor.created_at, cursor.id) ORDER BY created_at DESC LIMIT 10

Example: input-validation & Prototype Pollution

• The Challenge: Implement a recursive configuration loader that merges user-supplied JSON overrides with default system settings.
• The Trap: The model writes a standard recursive deepMerge(target, source). It validates basic types (strings, numbers, arrays) but omits key sanitization.
• The Hidden Invariant: The hidden test suite attempts a prototype pollution attack by sending a payload containing {"__proto__": {"isAdmin": true}}. If the candidate did not explicitly instruct the model to sanitize __proto__ and constructor, the hidden security test fails immediately.

3. The Anatomy of Hidden Test Grading

How do you grade trap tasks objectively without exposing the answers to the candidate?

In Plaiback’s architecture, tasks are divided into two physical directories:

1. starter/: The files the candidate receives (starter code, README, package configuration, and happy-path unit tests).
2. tests/: The platform’s private hidden test suite.

Task Structure:
├── task.json          (timebox, budgetCredits, runner config)
├── README.md          (The true business contract and specification)
├── starter/           (Seeded to candidate's container at session start)
│   ├── src/
│   │   └── ledger.ts
│   └── test/
│       └── basic.test.ts
└── tests/             (Held back; overlaid ONLY at grading time)
    └── hidden/
        ├── concurrency.test.ts
        ├── edge-cases.test.ts
        └── security.test.ts

When the candidate completes their work, the platform freezes the sandbox, copies the candidate’s final workspace to an isolated test container, overlays the tests/ directory, and runs the objective test suite (grade.ts).

The resulting pass/fail tally (grade.json) provides an indisputable ground truth for correctness that anchors the LLM judge’s qualitative rubric.

4. The 4 Difficulty Knobs for Challenge Tuning

When designing or customizing a task for your team, you can fine-tune difficulty by adjusting four core knobs:

1. Credit / Token Ceilings (budgetCredits): Restricting the API spend from $5.00 down to $0.50 punishes brute-force prompt-and-pray loops and forces prompt economy.
2. Timebox Windows (timeboxMinutes): Shortening the session from 60 minutes to 30 minutes tests rapid problem decomposition and high-bandwidth debugging.
3. Constraint Walls: Adding explicit environmental constraints in the README (e.g., “Zero external npm dependencies”, “Must execute in O(n) linear time”, “Do not use Node.js Intl”).
4. Specification Ambiguity: Providing an explicit interface contract vs. forcing the candidate to design their own domain models and characterization test harness.

5. Authoring Your Own Tasks in Plaiback

Whether you are running quarterly team katas or evaluating senior candidates, bringing your own codebase context produces the highest possible evaluation signal.

With Plaiback, creating a new challenge is as simple as running:

npm run new-task payment-reconciliation "Payment Reconciliation Engine"

This stamps a known-good skeleton with manifest validation, starter scaffolding, and hidden test hooks. You can author tasks directly in the web admin interface, import existing private Git repositories, or adopt pre-calibrated challenges directly from the Plaiback Task Catalog.

Stop testing whether engineers can remember syntax that LLMs can generate in milliseconds. Start testing whether your engineers have the judgment to steer, constrain, and verify those models when it matters most.

Explore our full catalog of pre-calibrated trap tasks: See Plaiback Kata or schedule a platform walkthrough.