Skill Skill Levels
Skills are way to save AI prompts for reuse. It's a simple concept hiding surprisingly deep capability. This post explains how to use skills at several levels of sophistication. I use Claude for examples, but the patterns apply to most chat agents.
Skill Skill Levels
Level 0: Write a Skill by Hand
At the most basic level, Skills are a text file containing a prompt. They're functionally the same as keeping a prompt notebook and copy/pasting prompts into a chat as needed. Skills provide a shortcut for this: Claude watches for you to type a / followed by a skill name (e.g. /my-skill) then it pastes the my-skill prompt into your chat behind the scenes. That's it.
The simplest way to create a skill is to type it directly into Claude. Open the "Customize" menu ➡️ skills ➡️ + . You'll get a window to type in your skill.
Alternatively, you can create a new text file, save it as SKILL.md and then import it. This naming convention helps your agent recognize the Skill and nudges text editors to apply Markdown conventions, which helps as your skills get more complex. You should also add some basic metadata at the top, which winds up looking like this:
| SKILL.md | |
|---|---|
1 2 3 4 5 6 7 | |
Skills are very easy to share around the office because they're just text files. Share skills by email, chat, file share, whatever. You'll eventually want stronger governance (version tracking, official skills, etc.) but swapping SKILL.md files via email or chat is fine to start.
Skills can contain more stuff, like reference files and python scripts. That makes it more complicated to package them up. You don't have to worry about this until Level 3 or so. When you do, ask your agent for help.
Level 1: Vibe Code a Skill
But why write a skill by hand when there's a friendly AI right here? Meet /skill-creator, the skill-writing skill! This skill instructs the AI to interview you about the skill you want to create, draft it for you, and save it for future use.
This is particularly useful when you have examples of the task you're trying to turn into a skill, like a long-running chat. Claude can review your past work to find common patterns for the skill. It will ask you clarifying questions when needed. This works even better in a project where you have several lengthy conversations and a collection of files attached. More data = better results.
Read the Source
Anthropic posted The full content of /skill-creator on Github. Read through it to build a deeper understanding of how skills work.
Level 2: Intentional Iterative Improvement
The vibe-coded skill is an example of one-shot development. Yeah Claude asked you some questions, but fundamentally it wrote the skill in a single session. AI always gets better with more data and iteration, so how do we apply that to our skill?
The simplest option is few-shot learning, which just means running a few improvement cycles cycles with your existing AI agent. You provide a way for the AI to evaluate the skill's quality, then ask it to improve the skill.
Evaluations can be tricky, but they don't have to be. Simply providing a handful of "When I give you X, the skill should produce Y" examples is quite useful. LangChain found that this approach can double accuracy!
Try Different Models
The same skill will produce different behaviors depending on the model you use it with. Try creating a skill, improving it with few shot learning, then running it with a different model. How did the results change? Repeat the few-shot process with the new model. Now how does it behave? Compare the content of the skills that have been optimized for different models.
You can get this for free by using the skill over time. Each time you run the skill, provide feedback on the AI's performance. Then use the /skill-creator or similar to make the AI incorporate your feedback into the skill.
Level 3: Quantitative Optimization
Moving beyond few-shot learning requires a big step in technical complexity. You start writing evaluation programs that score LLM output numerically. This provides the feedback that the computer needs to iteratively improve the skill without direct supervision. Tools like PromptFoo and DSPy](https://dspy.ai) help implement these patterns.
You can also use the "LLM-as-Judge" pattern here. That means instead of writing a deterministic computer program to score the output, you send it to another LLM to evaluate instead. This shines when you have access to a high-quality, expensive model and need to optimizing a skill for a smaller, cheaper model.
You can take this stage quite far. John Lindquist's Imps use harness-specific optimizations and automated evolution to deliver extreme performance on narrowly defined tasks (e.g. the git CLI). They tightly control context, allowing small, fast models to focus. They use hooks to preprocess user messages and invoke specialized skill-model pairs without invoking the main model.
Don't get banned
This is very similar to how model distillation works. The big AI companies are very sensitive about this. They might suspend your account if they mis-identify your LLM-as-judge activity.
Level 4: Autonomous Evolution
The final frontier of skill development automates all the above into a continuous evolution cycle. You could write a better evaluation program or buy a higher quality LLM judge, but the big unlock is a stream of new examples to optimize against. Where can you get this? Your own chats!
Every invocation of the skill is an opportunity to collect feedback on its performance. The skill learns about edge cases and how to prioritize information. You learn what workflow you like and how the Skill can best support you.
Where Level 2 uses occasional manual feedback and Level 3 creates one time evals, Level 4 runs both of these continuously. Agents identify when your skills start underperforming, perhaps due to a model change or evolving requirements. They update evals and refine skills in the background.
Collaboration can really help here. If your whole team uses the same skill, then everyone can provide feedback to refine the skill. This significantly increases the data available to improve the skill and gives the team a common reference point.
Conclusion
These Skill Levels reflect the broader state of AI right now. Even hugely capable models benefit from clear instructions and iterative development. We're slowly moving from AI chatbots to more complex systems that can remember things over time, pull in new information, and refine their understanding of specific domains. None of it works particularly well yet. It's still very easy to spend sink a ton of time and money into sophisticated AI projects with nothing to show for it.
For most people, using skills at all (Level 0) will significantly improve their experience with AI system. Working up to Level 2 should provide meaningful benefits without too much complexity. Pushing beyond that is only worth it if you have meaningful tech skills and significant challenges to figure out.
In a future post, I'll explore some tooling that designed to help with these higher level challenges.