DongHyuk Lim Forward Deployed Engineer
ENKO

AI Systems / Education

Redesigning a production pipeline around a requirement it could not support

A customer wanted to embed our question-generation tool inside its own service. Their product required every passage to produce two linked questions, but our existing process generated one independent item per pass and had no concept of shared passage context. I explained what could be fixed immediately, separated what required architectural change, and committed to a three-week redesign. Working in a two-person team, I rebuilt the generation flow while directly owning the architecture plan, prompt engineering, retrieval improvements and GPT API testing.

Shipped to production · annual contract signed 2-person teamFebruary–December 2024LangChain · RAG · GPT
15s8s

Generation latency

5%1.5%

Failure rate

3weeks

Committed and delivered redesign

~800runs

GPT API tests behind the decisions

On this page
  1. 01Project snapshot
  2. 02What I owned
  3. 03The system before the change
  4. 04The new customer requirement
  5. 05Scoping the request with the customer
  6. 06Key decisions
  7. 07Architecture: before and after
  8. 08The three weeks
  9. 09Testing and performance
  10. 10Outcome
  11. 11What I learned
01

Project snapshot

CustomerB2B English education-content provider (anonymised)
ProductRAG-based English question-generation service
DurationFebruary–December 2024
Team2 people; re-architecture implemented together
My rolePlanning and engineering: customer scoping, architecture plan, retrieval and prompt improvements, GPT API testing
EnvironmentLangChain · RAG · GPT API · vector retrieval
StatusShipped; customer requirement delivered; annual contract signed
02

What I owned

I worked across the customer and engineering boundary rather than treating the new request as a prompt-only change.

BuiltDiagnosis of the recurring quality problems in retrieval and generation.
BuiltSplitting the customer’s request into changes possible immediately and changes requiring a new architecture — and explaining the difference to the customer directly.
BuiltPrompt engineering and roughly 800 GPT API test generations.
BuiltRetrieval selection, item structure, validation, token management and caching improvements.
Co-builtThe re-architecture: I owned the plan and implemented it with one teammate.
03

The system before the change

The service generated English questions from a passage using LangChain, retrieval-augmented generation and GPT. It was already in use, but quality had begun to drift.

  • Duplicate questions appeared in the output.
  • Incorrect options were sometimes unrelated to the passage.
  • Few-shot retrieval selected examples that did not match the intended question type or student level.
  • Failures were not consistently caught before reaching the final output.

These were not isolated prompt defects. Retrieval, output structure and validation all contributed to the final quality.

04

The new customer requirement

The customer planned to embed the tool into its own service, where a one-passage, two-question format was a core product requirement.

“Generate two related questions from the same passage.”

Customer requirement

The request sounded like an additional output option, but the system was organised around one independent item per generation pass. Each item managed its own context, and the pipeline had no shared passage object that two questions could reliably reference.

Adding another prompt call would create two questions, but it would not guarantee that they shared the same passage context or formed a valid linked set.

Visible requestGenerate two questions instead of one.
Architectural gapThe pipeline had no passage-level context shared across multiple generated items.
Engineering problemRedesign generation, token management, caching and validation around the passage rather than the individual question.
05

Scoping the request with the customer

I divided the request into two groups and explained the difference directly.

Could be improved immediately

  • Retrieval selection and few-shot quality
  • Prompt and output consistency
  • Item formatting and validation rules
  • Existing single-item quality defects

Required architectural change

  • Shared passage context across multiple questions
  • Separation of generation stages
  • Passage-level token management
  • Cache behaviour for linked outputs

A limitation paired with a date

I told the customer that the current architecture could not support the complete requirement reliably, then committed to delivering the structural changes within three weeks. The conversation established a clear boundary without ending at “no”.

06

Key decisions

01Separate immediate quality fixes from structural work

Treating every issue as one large rewrite would have made progress difficult to explain and verify. I separated retrieval, prompt and validation improvements from the architecture required for linked questions.

This gave the customer a clear view of what could improve now and what depended on the redesign.

trade-offTwo workstreams to manage instead of one. Accepted: the customer could see progress before the redesign landed.
02Make the passage a first-class unit of the pipeline

The old system centred each generation pass on one question. The new architecture centred the workflow on a shared passage context that could coordinate multiple questions.

This allowed linked items to use the same source context while preserving their individual generation and validation steps.

trade-offToken management and cache keys both had to be rebuilt around the new unit. Accepted: it was the only structure that made a linked set verifiable.
03Improve few-shot selection by student level and similarity

The retrieval system could select examples that were semantically similar but educationally inappropriate. I retuned the vector values by school level and rewrote similarity selection so few-shot examples matched both the content and the intended learner.

This addressed a failure source upstream of the prompt instead of trying to repair every bad example after generation.

trade-offA narrower candidate pool per request. Accepted: fewer, better-matched examples beat more, wrongly-levelled ones.
04Standardise item structure and add multi-stage validation

I standardised the representation of generated items and introduced multiple validation stages. This made duplicate, relevance and structure failures easier to detect before the final response.

Validation became part of the generation system rather than a final manual check.

trade-offAdditional latency inside the pipeline. Absorbed — the redesign still cut end-to-end time from 15s to 8s.
05Measure behaviour across repeated GPT runs

Individual examples were too variable to support architecture decisions. I ran approximately 800 GPT API tests across the changed pipeline and used repeated results to refine prompts, retrieval and validation.

trade-offTest cost and time. Accepted: the alternative was tuning a stochastic system on anecdotes.
07

Architecture: before and after

One item per isolated pass → passage-centred staged generation
One item per isolated pass → passage-centred staged generation — BeforeInput passageVector retrievalSingle-pass generationOne independent questionOutputno shared passage state
One item per isolated pass → passage-centred staged generation — AfterLevel + similarity retrievalInput passageShared passage contextpassage-level token managementGeneration stage 1Generation stage 2Linked question 1Linked question 2Multi-stage validationReworked cacheFinal linked set
Before, the system had no shared passage-level state, so producing a second item meant running another isolated generation path. After, the pipeline manages context and tokens at the passage level, separates generation stages, validates each item as part of a linked set, and caches the resulting structure consistently.

Stage names are generic here. Actual pipeline component and cache names are withheld until the customer confirms what can be disclosed.

08

The three weeks

Before week 1

Reproduced the failures

Duplicate items, unrelated distractors, and few-shot examples selected at the wrong level — reproduced rather than reported.

Before week 1

Classified the request

Immediate fixes separated from architecture-dependent requirements, then explained to the customer with a three-week commitment.

Week 1

Retrieval rebuilt

Vector values retuned by school level; similarity selection rewritten so examples matched content and learner.

Week 1–2

Structure and validation

Generated item structure standardised; multi-stage validation added inside the pipeline.

Week 2

Passage-centred redesign

Shared passage context, separated generation stages, reworked passage-level token management and cache behaviour.

Week 2–3

~800 GPT API tests

Repeated runs across the changed pipeline; prompts, retrieval and validation refined from the aggregate, not from single outputs.

Week 3

Shipped

The linked-question requirement delivered within the committed timeline and deployed to the live service.

09

Testing and performance

Measured before and after the redesign
Generation latencyend to end, per request−47%
Before15s
After8s
Failure rategenerations rejected or unusable−70%
Before5%
After1.5%

Figures describe the tested generation pipeline after the changes. Exact measurement conditions and the distribution of the ~800 test runs across question types are pending customer confirmation before publication.

10

Outcome

Measured results

  • Generation latency improved from 15 seconds to 8 seconds.
  • Failure rate decreased from 5% to 1.5%.
  • The previously unsupported linked-question requirement was delivered within three weeks.
  • The completed work contributed to a USD 40K annual B2B contract.

Delivery status

  • The redesigned pipeline was shipped to the live service.
  • The customer requirement was completed within the committed timeline.

Scope of the claim

  • Performance figures describe the generation pipeline after the changes, measured internally.
  • The re-architecture was implemented by two people; the plan, prompts, retrieval work and testing were mine.
11

What I learned

Technical honesty was most useful when paired with a delivery path. Dividing the request into “possible now” and “possible after architectural change” let me explain a real limitation without turning it into a dead end for the customer.

The quality problems also reinforced that an LLM output failure may begin before the prompt. Retrieval quality, shared context, structure, validation and caching all shaped the final result.

If I repeated the project, I would add explicit retrieval-quality checks earlier. That would have caught the few-shot mismatch before it became a downstream generation problem.

Diagrams are drawn from the real systems and redrawn here; customer names, data and identifiers are removed.