The AI Product Maturity Assessment

TL;DR

• AI product maturity is a 5-level model: Capable (works correctly), Adaptive (learns from patterns), Personal (understands individuals), Anticipatory (predicts needs), and Aligned (maintains a living model of each user)
• 78 percent of AI products are stuck at Level 1 because the jump to Level 2 requires user modeling infrastructure that standard ML architectures do not include
• Each level requires specific architectural capabilities and produces measurable improvements in retention, satisfaction, and user lifetime value

AI product maturity is a five-level model measured by depth of user understanding, not model capability: Capable, Adaptive, Personal, Anticipatory, and Aligned. 78 percent of AI products are stuck at Level 1 (capable but generic) because the jump to Level 2 requires user modeling infrastructure that standard ML architectures do not include. This post covers the five maturity levels with concrete criteria, the architectural requirements for each level, and how to assess and advance your product’s maturity.

The Five Levels

Level 1: Capable

The AI works. It produces correct, useful output for generic queries. It handles the common cases well and fails gracefully on edge cases. From a model perspective, it is well-trained, properly evaluated, and performant.

But it treats every user the same. A first-time user and a year-long power user receive identical experiences. The AI has no memory, no preferences, no understanding of who is using it. Every session starts from zero.

78 percent of AI products are at this level. They have invested heavily in model capability and almost nothing in user understanding. Their retention curves show the classic AI pattern: strong first-week engagement, steep drop-off by week 4.

Level 2: Adaptive

The AI adjusts its behavior based on observable patterns. It notices that a user prefers concise responses and starts producing shorter output. It detects that a user frequently asks about Python and starts defaulting to Python examples. The adaptation is behavioral, based on what the AI observes, not what it understands.

Level 2 products use session history, preference tracking, and basic behavioral signals to modify AI behavior. The improvement in user experience is noticeable, the product feels less generic. But the adaptation is shallow. It catches surface-level patterns without understanding underlying needs.

About 15 percent of AI products reach Level 2. The architectural requirement is a preference storage system and the ability to condition AI responses on stored preferences.

Level 3: Personal

The AI understands each user as an individual. It knows their goals, their constraints, their communication style, their domain expertise, their evolving needs. The understanding is not just behavioral (what they do) but epistemic (what they believe and know).

Level 3 products have self-models, structured representations of each user that capture beliefs, preferences, and context. The AI does not just adapt to patterns; it understands the person behind the patterns.

About 5 percent of AI products reach Level 3. The architectural requirement is a self-model system with belief tracking, confidence scoring, and continuous updating.

Level 4: Anticipatory

The AI predicts what users will need before they ask. It notices that a user’s beliefs are shifting and proactively surfaces relevant information. It detects that a project deadline is approaching and adjusts its behavior to match the urgency. It anticipates needs based on deep understanding of the user’s context and trajectory.

Level 4 products use self-models predictively, not just to personalize responses, but to anticipate needs and take proactive action. The user experience shifts from “the AI responds well” to “the AI knows what I need.”

Less than 2 percent of AI products reach Level 4. The architectural requirement is predictive self-model inference and proactive interaction capabilities.

Level 5: Aligned

The AI maintains a living model of the user that evolves in dialogue, the user can see, validate, and correct the model. The relationship between the user and the AI is transparent, consensual, and genuinely bilateral. The AI is not just personal; it is aligned, its understanding matches the user’s self-understanding.

Level 5 is the aspiration. No current AI product fully achieves it. The architectural requirement is transparent self-models, user-facing model inspection, and bilateral model evolution.

The Assessment Framework

For each level, there are specific capabilities to evaluate. Here is the assessment framework I use when evaluating AI products.

1// AI Product Maturity Assessment← Self-assessment framework
2const assessment = {
3  level1_capable: {
4    criteria: ['Model accuracy > 85%', 'P95 latency < 500ms',
5      'Graceful error handling', 'Consistent output quality'],
6    score: null, // assess each criterion: 0-1
7  },
8  level2_adaptive: {
9    criteria: ['Session preference tracking', 'Behavioral pattern detection',
10      'Response style adaptation', 'Cross-session preference persistence'],
11    requires: 'preference storage + conditioning',
12  },
13  level3_personal: {
14    criteria: ['Structured self-models per user', 'Belief tracking with confidence',
15      'Goal and context awareness', 'Evolving understanding over time'],
16    requires: 'self-model infrastructure (e.g., Clarity API)',
17  },
18  level4_anticipatory: {
19    criteria: ['Predictive need detection', 'Proactive suggestions',
20      'Context-aware timing', 'Trajectory-based personalization'],
21    requires: 'predictive self-model inference',
22  },
23};

Why Teams Get Stuck at Level 1

The jump from Level 1 to Level 2 is the hardest because it requires a fundamentally different architectural mindset.

Level 1 is a model problem. You invest in better training data, better fine-tuning, better prompt engineering. The feedback loop is clear: improve the model, improve the output, improve the metrics.

Level 2 is a user problem. You need to capture, store, and act on individual user information. This requires infrastructure that standard ML pipelines do not include: preference storage, behavioral tracking, conditional response generation. The model is no longer the only variable, the user model matters too.

Most teams try to reach Level 2 by improving their Level 1 capabilities, better models, better prompts, more training data. This does not work because the limitation is not model quality. A Level 1 product with a perfect model is still a generic product. The user does not care how good your model is if it treats them like a stranger.

The architectural shift is conceptual as much as technical. The team must move from “how do we make the AI smarter?” to “how do we make the AI understand each user?” Those are different questions with different answers and different infrastructure requirements.

Trade-offs

Maturity levels are descriptive, not prescriptive. Not every product needs to reach Level 5. A search engine at Level 1 can be perfectly successful. A health coaching app at Level 1 is inadequate. The right maturity target depends on the product’s relationship depth with users.

Higher maturity levels require more data. Level 3 and above require significant per-user data to build effective self-models. Products with low engagement frequency may struggle to accumulate enough observations. The mitigation is focusing self-model investment on high-engagement user segments first.

The assessment can be gamed. A product could technically implement preference tracking (Level 2) without meaningfully improving user experience. The assessment should be validated against outcome metrics, retention, satisfaction, alignment scores, not just feature checkboxes.

What to Do Next

1. Assess your current level. Use the framework above to evaluate your AI product against each level’s criteria. Be honest, most teams overestimate their maturity by one level. If you are uncertain between two levels, you are at the lower one.

2. Identify the blocking capability. For your current level plus one, identify which specific capability you lack. The blocker for Level 2 is usually preference persistence. The blocker for Level 3 is usually self-model infrastructure. The blocker for Level 4 is usually predictive inference. Focusing on the specific blocking capability produces faster maturity improvement than general investment.

3. Set a 6-month maturity target. Advancing one maturity level takes 3-6 months of focused investment. Set a target level, define the specific capabilities needed, allocate engineering resources, and measure progress through the retention improvement that each level produces. The retention delta between levels is the ROI justification for the investment.

References

1. Product vs. Feature Teams
2. only 1 in 26 unhappy customers actually complains
3. Qualtrics notes in their churn prediction framework
4. Continuous Discovery Habits
5. 80% of features in the average software product are rarely or never used

Related

• Minimum Viable Evals: Start Here
• LLM-as-Judge: Using AI to Grade AI
• The AI Product Health Check