Product overview

How Litmus works.

Hire engineers who last.

What Litmus does

Litmus generates and hosts coding interviews built from your codebase, tickets, and job descriptions. Instead of generic OAs and take-homes, candidates build a real feature with their usual tools while we track file iteration, commit history, and AI usage. Every submission runs against a grading harness in a sandbox, and you get an evidence-based report on how each engineer builds.

Where Litmus fits in

Litmus can fit in anywhere in your hiring pipeline. Here are a few places other companies have found Litmus useful:

ApplicationRecruiter screenOnsiteOfferFirst touchTop of funnelDeep signal
  • First touch · Submitted with the application: a technical read before a resume even reaches a human.
  • Top of funnel · The most common placement: narrow a large inbound pool before anyone spends onsite or oncall time.
  • Deep signal · For work trials and extended projects: granular tracking across a long, complex task.

What changes for your team

Engineer time

Zero

engineering hours to run your top of funnel

No designing interview problems, no screening calls, no grading early submissions. By the time an engineer meets a candidate, they've already built in your context.

Your workflows

Plug-in

to the process you already run

Candidates pass to and from your ATS automatically, or you can manage them start to finish on Litmus. Nothing about your pipeline has to change.

Signal

Work-trial

depth from every submission

How someone builds across a long task, how they use AI in practice, how they explain their decisions. None of it shows up in a standard interview.

How assessments are built

Host an existing assessment you love, or generate one from scratch by linking your repos, tickets, and job descriptions. Test candidates on real end-to-end feature work to see how they build. Engineers solve problems like they would on the job, not just implement solution specs.

Build me an assessment from this ticket

ENG-482Bulk export pipeline

Litmus building now · ~2 minutes

ReadENG-482 + your stack
WriteREADME.md
Scaffoldexports/ · 14 files
Harness6 graded requirements
Questionswalkthrough · 4
Exports150 min · TypeScriptReview

Real assessments built this way.

Marketgraded belowA prediction-market engine with an automated market maker, graded under concurrent trades.
CalA booking engine that survives DST transitions and concurrent hold stampedes.
BlocksA paged KV-cache scheduler for LLM serving: prefix sharing, copy-on-write, preemption.
FlashA streaming attention kernel: online softmax, no N×N matrix, stable at large magnitudes.

What the candidate experience looks like

01

Initialize

Setup is two terminal commands. The assessment pulls straight into the candidate’s own workspace.

candidate@laptop: ~ — zsh

$ npm install -g litmus-cli

+ litmus-cli@1.3.9 installed

$ litmus init 7f2a…

  Market

  Time limit: 150 minutes

Connected

Created market/

✓ Assessment ready

  Next steps:

    cd market

    code .  or  cursor .

  Your work is being tracked. When you're done: litmus submit

market — Visual Studio Code
lmsr.tsREADME.md
ProblemsOutputTerminal

claude · Sonnet 4.6

> hold the per-market lock across read and write

editing app/trade/route.ts…

Claude CodeCodex
main✓ 0  ⚠ 0litmus tracking
candidate@laptop: ~/market — zsh

$ litmus submit

  Submitting: Market

  Upload size: 1.8 MB (38 files)

  Tracking log included

  Confirm submission? [Y/n] y

Submission received

✓ Code submitted — one step left.

  litmushiring.com/candidate/market/record

Assessment folder deleted: market/

  You may need to `cd` to another directory.

litmushiring.com/candidate/market/record
Market24:0601:42
lib/lmsr.ts

$ npm run smoke

52/52 checks passed · 1.9s

Recording in progressMicSubmit Walkthrough

How grading works

Grading is tailored to each assessment and designed to surface signal on how an engineer would actually build on your team, forming a ranking score across different assessment dimensions.

Run by the harness

Hard requirements execute in a live sandbox

CompilationConcurrencyCorrectness

Read by graders

Qualitative dimensions, judged with evidence

Code qualityAI usage
Both land in one evidence-taggedE7E7 report
litmushiring.com · submission review
Walkthroughrecorded during the assessment

Final submission

  • app
  • audit
  • route.ts
  • markets
  • [id]
  • resolve
  • route.ts
  • trade
  • route.ts
  • route.ts
  • route.ts
  • users
  • [id]
  • route.ts
  • globals.css
  • layout.tsx
  • page.tsx
  • data
  • seed_markets.json
  • lib
  • engine.ts
  • errors.ts
  • http.ts
  • lmsr.ts
  • state.ts
  • scripts
  • smoke.mjs
  • tests
  • CONTRACT.md
  • README.md
  • api.test.mjs
  • run.mjs
  • DESIGN.md
  • README.md
  • next-env.d.ts
  • package-lock.json
  • package.json
  • tsconfig.json
  • tsconfig.tsbuildinfo
  • types.ts
README.mdPreviewCode

Market

A prediction-market engine with an automated market maker.

Overview

Prediction markets (Polymarket, Kalshi, the old internal markets at every big tech company) turn opinions into prices: people buy shares in "YES" or "NO" on some outcome, the price moves with demand, and when the event resolves, each winning share pays out one dollar. The clever part is the market maker. Instead of matching buyers to sellers, an automated market maker (the standard one is Hanson's LMSR) always quotes a price from a cost function, so there is always liquidity and the math is fully determined: every trade has exactly one correct price, cost, and resulting state.

You're going to build the engine. Users trade YES/NO shares against an LMSR market maker, prices update, balances and positions track, and on resolution the winners get paid. It has to be exactly right, and it has to stay right when trades land concurrently on the same market.

Problem Statement

Build the backend and a minimal trading UI for binary prediction markets run by an LMSR market maker. Support creating markets, quoting prices, buying and selling shares (priced by the cost function), tracking per-user balances and positions, and resolving a market so winning shares pay out. Trades on the same market must be atomic: concurrent trades must produce exactly the state that applying them one at a time would, with no lost updates, no negative balances, and no overselling.

The market maker (LMSR)

Each market has a fixed liquidity parameter b (provided). Let q = (q_YES, q_NO) be the net shares outstanding of each outcome.

Cost function:  C(q) = b * ln( exp(q_YES / b) + exp(q_NO / b) )
Price:          p_i  = exp(q_i / b) / ( exp(q_YES / b) + exp(q_NO / b) )
Cost to trade d shares of outcome i:   C(q with q_i += d) - C(q)
                                       (a negative cost is a payout, i.e. a sell)

Prices always sum to 1. Implement these directly; the engineering is the test, not deriving the formula.

Getting Started

Prerequisites

  • Node.js 20+
  • Any modern framework (Next.js, Vite + React, SvelteKit). Starter is Next.js.

Setup

Dependencies are installed automatically when you initialize the assessment with the Litmus CLI. You're ready to start coding.

Files in the workspace:

  • types.ts defines Market, User, Trade, and Outcome.
  • data/seed_markets.json has a couple of seed markets and a deterministic example trade sequence to develop against.
  • tests/CONTRACT.md is the API contract the grader drives, including the invariants it checks.

Requirements

  1. Expose an HTTP API (see tests/CONTRACT.md): create a market, get its prices, trade, get a user's balance and positions, and resolve.
  2. LMSR correctness: GET prices equal softmax(q / b) and sum to 1; a trade's cost equals C(q') - C(q) exactly. Each user starts with a fixed balance (provided).
  3. Trading: buying d shares of an outcome charges C(q') - C(q) and increases the position; selling (negative d) refunds and decreases it. Reject a trade that would overdraw the balance or sell more shares than the user holds. Balances never go negative; positions never go below zero.
  4. Atomicity: concurrent trades on the same market must apply atomically. If N trades on one market land at once, the final q, prices, balances, and positions must equal the result of applying them sequentially. No lost updates.
  5. Resolution: resolve(outcome) pays each holder of the winning outcome 1.0 per share into their balance, pays losing shares 0, clears positions, and closes the market to further trades.
  6. Accounting integrity: no funds are created or destroyed beyond the market maker's bounded subsidy (LMSR's max loss is b * ln 2 for a binary market). Document your accounting.
  7. A minimal trading UI: current prices, the user's positions, buy/sell, and a resolution view. The grader drives the API; keep the UI simple but real.

Examples

Example 1: Price after a buy

POST /markets { question: "Will it rain tomorrow?", b: 100 }   -> { market_id, prices: { YES: 0.5, NO: 0.5 } }
POST /markets/{id}/trade { user_id: "u1", outcome: "YES", shares: 50 }
  -> cost = C([50,0]) - C([0,0]); YES price rises above 0.5; prices still sum to 1

Example 2: Concurrency

20 buys of YES on the same market, fired concurrently.
Final q_YES == 20 * shares, and final prices/balances == applying the 20 trades one at a time.

Example 3: Resolution

POST /markets/{id}/resolve { outcome: "YES" }
  -> every YES share pays 1.0 into the holder's balance; NO shares pay 0; further trades rejected.

Submission Guidelines

What to Submit

  • All source code (backend, frontend, anything else you build).

How to Submit

litmus submit

Implementation notes

DESIGN.md is the full design and the reasoning behind each choice. This section is the short version: how to run it, how it's laid out, and how the accounting works (Requirement 6).

Running

npm run dev                    # http://localhost:3000 — API + UI
npm run smoke                  # 52 assertions against a running server
npm run build && npm start     # production build (routes must work here too)

The grader hits the API at the root path (POST /markets, not /api/...). The UI is at /. GET /audit is an extra endpoint — the executable version of the accounting below.

Layout

FileRole
lib/lmsr.tsPure LMSR math — cost, prices, tradeCost. Imports nothing, so the UI reuses it for a live cost preview.
lib/state.tsThe one globalThis-pinned store, and the global lock (withLock) that lives inside it.
lib/engine.tsThe only module that mutates state. Every workflow, and the audit.
app/**/route.tsSix thin HTTP handlers: parse → call engine → map errors to status codes.
app/page.tsxThe trading UI.
scripts/smoke.mjsZero-dependency test harness with its own reference LMSR.

Atomicity in one paragraph

Every state change goes through one global lock whose critical section is strictly synchronous (withLock<T>(fn: () => T) — the type forbids an await inside). Node runs that block to completion without interleaving, so a read-modify-write is atomic. The lock is global rather than per-market because a user's balance is shared across markets, and a per-market lock could not protect it. The order inside is READ → SCREEN → PRICE → AFFORD → COMMIT → SNAPSHOT: affordability is checked after pricing (you can't know if a trade overdraws until you've priced it) and inside the lock (checking before is the TOCTOU bug that lets two concurrent buys both pass a stale balance check). A rejected trade throws before COMMIT, so it leaves the world byte-for-byte unchanged.

Accounting (Requirement 6)

Where money comes from and goes. Every user is minted with 1000 on first reference. A buy of d shares moves C(q') − C(q) from the user to the market maker; a sell moves it back (a negative cost is a refund). On resolution, the maker pays each holder of the winning outcome 1.0 per share.

What the maker collected is derived, not accumulated. For a market that opened at q = 0, the total the maker has collected is exactly

collected = C(q_final) − C(0)

because trade costs telescope: Σ [C(qₖ) − C(qₖ₋₁)] = C(q_final) − C(q_0). The audit computes collected straight from the current q rather than keeping a running sum — a running total can drift from the truth by accumulated float error; a value derived from q cannot. (The smoke harness keeps an independent running sum and checks the two agree — they do, to relative error < 1e-12.)

The subsidy is bounded by b · ln 2. On resolution with winner W, the maker's net is collected − payout = [C(q_f) − C(0)] − q_W, so its worst-case loss is

payout − collected = q_W − C(q_f) + b·ln2  ≤  b·ln2

since C(q) = b·ln(Σ e^{qᵢ/b}) ≥ b·ln(e^{max(q)/b}) = max(q) ≥ q_W. The bound is tight — equality is the worst case — so /audit checks subsidy ≤ b·ln2 + 1e-6 rather than a strict , which one ulp of float noise would fail on a correct engine.

Global conservation, checked live by GET /audit:

Σ balances  ==  1000 · (users minted)  −  Σ collected  +  Σ payouts

This holds no matter the interleaving of concurrent trades, because it depends only on the final q (order-invariant) and the mint count — not on who traded in which order. /audit returns both sides, the delta (0 on a correct run), each market's subsidy against its ceiling, and the q == Σ positions check for every open market.

One invariant underwrites the rest. A user can only sell shares they hold, and qᵢ moves by the same d as their position, so qᵢ == Σ_users positions[market][i] at all times a market is open. That single identity is what guarantees positions never go below zero, q never goes negative, and resolution can pay out q_W knowing it exactly matches the shares people hold.

Deliberate scope decisions

  • In-memory, single process. The contract needs no persistence. This is correct under next dev / next start (one process); it would need a real serializable store behind a cluster or serverless — the withLock seam is where BEGIN … SELECT FOR UPDATE … COMMIT would go. The store logs its pid once so a stray second process is visible rather than silently wrong.
  • No slippage protection. A quote can move against you if another trade lands first. Real venues take a max_cost; the contract doesn't ask for one.
  • Rejections are all-or-nothing — an overdraw is rejected, not partially filled, matching the contract's "reject a trade that would overdraw."
  • shares: 0 is a 200 no-op (cost: 0), not a rejection — it violates no invariant, and the contract only mandates rejecting overdraw and oversell.

Elena Zhao

80/ 100
Share ReportView Submission
Strong Pass
Strong Pass: perfect harness scores across pricing, invariants, and concurrency; clean LMSR layering in ~40 minutes (, , ).

Code installs and runs; 1 of 1 checks passed.

Engineering profile
What they built
Elena built a Next.js prediction-market API with six REST routes under app/markets and app/users, backed by an in-memory state store in lib/state.ts that serializes per-market writes through a mutex. The LMSR cost function lives entirely in lib/lmsr.ts and is called by lib/engine.ts, which handles trade execution, balance updates, and resolution; thin route files in the app directory do only HTTP parsing and response shaping. A 52-check test suite in tests/api.test.mjs covers pricing math, invariant preservation, concurrent burst correctness, and overdraw rejection.
Code:+5,909 / -88
Commits:5
Session:40 min
AI tools:claude
AI usage
Claude
Claude04:44 AM
can you read through the readme and get context on what this assessment requirements are. i want to start by specing out a design architecture that focuses on completing the core requirements and goals we need to implement in detail.
Claude04:46 AM
send multiple sub-agents that verify that this plan is solid across the following domains: 1. basic market actions: can markets get created, can prices get quoted, buying and selling, tracking balances and positions for each user, resolving a market 2. testing for atomicity: depl…
Claude05:04 AM
what happens under this architecture if someone tries to buy at a price that they don't have in their balance?
Claude05:10 AM
cool let's implement. commit often
Harness
OKjson
ranTrue
errorNone
api base'(root)'
strategy'next-start'
pricing score100.0
invariants score100.0
concurrency score100.0
concurrency consistentTrue
rejects state unchangedTrue
endpoints reachable100.0
detail{'root': '/home/user/app'
Dimension breakdown
What stood out
  • Harness sweep: 100/100 on pricing, invariants, and concurrency across 71 checks ().
  • LMSR isolation: math confined to lib/lmsr.ts with log-sum-exp guard; engine separate from routes (, ).
  • Concurrency correctness: per-market mutex in state.ts; burst bursts conserve q and revenue exactly (, ).
  • Written design docs: DESIGN.md explains locking tradeoff and subsidy accounting with specifics (, ).
What gave us pause
  • Walkthrough thin: transcript is a placeholder; no live explanation of math or design choices ().
  • Overflow path unconfirmed: log-sum-exp guard documented but not stress-tested at extreme quantities ().
  • No live question responses: 0 of 0 questions answered; depth evidence is entirely static ().
Interview questions
  • lib/lmsr.ts uses log-sum-exp; walk through what breaks without it and how your implementation handles b-scaling at extremes (, ).
  • DESIGN.md justifies per-market over global lock for throughput; what failure mode emerges if two markets share a user balance? (, ).
  • The smoke harness fires 52 checks (); which cases were hardest to get right and what edge did you miss first?
  • Resolution maker_loss is 21.9 against a subsidy bound of 69.3 (); explain the accounting guarantee and where it could break.

Integrations

GitHubLinearJiraNotionGreenhouseAshbyLever

See it end to end.

Watch the demo

Questions? Email us at founders@litmushiring.com