# Skill: Choose between regex and LLM for text parsing, and build a cost-saving hybrid pipeline

> Apply this skill before parsing any text corpus -- it routes structured repeating text to cheap regex rules and reserves the LLM only for the ambiguous remainder, cutting costs by up to 95% versus routing everything through the model.

Source: sodigi·learn -- vibecoding/regex-ili-neyroset · https://sodigi.io/learn/vibecoding/regex-ili-neyroset

## When to use
- Any time the task involves extracting structured data from a text file or stream.
- Before routing a large batch of records through an LLM -- check first whether the format is uniform enough for rules.
- When a parsing pipeline is running slowly or burning budget unexpectedly.
- When reliability matters: for uniform text, regex gives the exact same answer every time; an LLM occasionally invents.

## Core rules
- First question, always: is the text tidy (90%+ follows one template) or chaotic (free-form, different every time)?
- Tidy text goes to regex rules first. Only send leftovers to the LLM.
- Chaotic or free-form text goes directly to the LLM -- regex cannot handle variable structure.
- The hybrid pipeline is the strongest default: rules cover 95-98% of the stream; the LLM handles only the flagged leftovers.
- After the rule pass, run a confidence check: flag records that parsed poorly (too few fields, missing answer, suspiciously short text). Send only flagged records to the LLM.
- For the LLM fallback, use the cheapest capable model -- a small/Haiku-class model is sufficient for a handful of hard cases.
- Never trust "the rule seems to have worked" without a count. Ask for: how many records parsed, how many were flagged.

## Procedure
1. Inspect a sample of the text. Scroll through 20-30 records. Are they structurally identical?
2. If yes (tidy): write a regex that extracts the needed fields. Test it against the full corpus.
3. Run the confidence check: count records where the extracted result is incomplete or suspicious.
4. If the pass rate is 95%+: done, no LLM needed.
5. If the pass rate is below 95%: collect the flagged records and send only those to the LLM with a targeted extraction prompt.
6. Merge the rule-parsed results with the LLM-patched results.
7. Report the final split: what percentage the rules handled vs. how many the LLM had to fix.

## Ready-to-use prompt
```
I have a file with [N] records. They all share this format: [describe the
repeating structure].

Do this:
1. Write a regex rule that pulls [the fields you need] from this tidy text.
   Run it over the whole file.
2. Count which records parsed poorly: missing fields, suspiciously short
   content, or incomplete structure. Show me the list.
3. Parse only those flagged records intelligently, by meaning.
   Leave the well-parsed records alone.
4. At the end, tell me: what percentage the rule handled and how many records
   had to be patched up by the LLM.
```

## Pitfalls
- Routing everything through the LLM "just to be safe" when the text is uniform -- this is like hiring a professor to count tally marks. Expensive, slow, and unnecessary.
- Forcing regex onto genuinely free-form text (handwritten notes, varied-structure emails) -- the rule will fail on edge cases and produce silent data loss.
- Skipping the confidence check and blindly trusting the rule output -- some records will parse incorrectly and no one will know.
- Using an expensive frontier model for the LLM fallback -- a cheap small model is more than enough for a handful of hard cases.
- Not testing on messy edge cases before the full run -- empty lines, missing fields, and unusual characters are exactly where regex breaks.
- Treating the choice as permanent -- the same pipeline can mix both: rules for the clean majority, LLM for the messy tail.