Observation Contexts Explained

TL;DR

• Observation contexts are structured dimensions that define what a product observes about each user - turning raw interaction data into meaningful, actionable understanding
• Without observation contexts, self-models become unstructured data stores that accumulate information without framework for interpretation, leading to poor personalization despite rich data
• Products with well-defined observation contexts achieve meaningful personalization 3x faster because they know exactly what to look for and how to interpret what they find

Observation contexts are structured dimensions that define what a product observes about each user, turning raw interaction data into meaningful, confidence-weighted understanding. Without them, self-models become unstructured data stores that accumulate hundreds of attributes with no framework for interpretation. This post covers how to design observation contexts, define state progressions, map behavioral signals, and avoid the common mistakes that lead to data-rich but insight-poor personalization.

What Observation Contexts Are

An observation context is a defined dimension along which a product observes and understands a user. It is not a data point. It is a lens - a specific way of looking at user behavior that extracts meaning.

Think of it like this. If a self-model is a portrait of a user, observation contexts are the specific aspects of that portrait: face shape, expression, posture, clothing, background. Without defined aspects, you are just collecting random pixels. With defined aspects, every pixel contributes to a coherent understanding.

Each observation context has four components.

The dimension. What are you observing? This is the named aspect of user understanding. Examples: communication style preference, domain expertise depth, learning velocity, risk tolerance.

The state progression. What are the possible states within this dimension, and how do they relate to each other? For domain expertise depth, the progression might be: novice, familiar, competent, expert, authority. These states are ordered and directional - users typically progress through them, though they can move backward.

The confidence threshold. How much evidence do you need before you assign a state with confidence? This prevents premature categorization. You might need 5 observations to classify a user as novice with moderate confidence, but 20 observations to classify them as expert with high confidence.

The observation signals. What behavioral signals feed into this context? For communication style preference, signals might include: average output length accepted, frequency of output modifications, use of formatting features, response to bullet points versus paragraphs.

Designing Observation Contexts

The design of observation contexts is one of the most important architectural decisions in building a self-model system. Get this right and your product builds understanding efficiently. Get it wrong and you collect data that never becomes useful.

Here are the principles I have learned from designing observation contexts across multiple AI products.

Start with product decisions, not user attributes. The question is not what can we know about the user? The question is what decisions does our product make that should vary per user? Work backward from the personalization decisions to the understanding needed to the observation contexts that produce that understanding.

For an AI writing assistant, product decisions include: output length, tone formality, domain vocabulary, level of explanation, suggestion aggressiveness. Each of these maps to an observation context: brevity preference, communication formality, domain expertise, learning stage, workflow integration depth.

Limit contexts to 8-15. This is counterintuitive. More contexts means more granular understanding, right? In practice, more contexts means more noise, slower confidence building, and harder interpretation. The enterprise team I mentioned earlier improved personalization by reducing from 200+ attributes to 12 contexts. Constraints force clarity.

Define state progressions carefully. State progressions should be mutually exclusive, collectively exhaustive, and ordered. For domain expertise: novice, familiar, competent, expert. Not novice, intermediate, kind-of-expert, expert - those overlap. And not novice, expert - that misses the gradations that matter for personalization.

Map behavioral signals explicitly. For each context, document exactly which behavioral signals feed into it and how they are weighted. Communication formality might be informed by: vocabulary complexity (30 percent), sentence structure (25 percent), emoji usage (15 percent), greeting patterns (15 percent), formatting choices (15 percent). Make the mapping explicit so you can debug and improve it.

1// Define an observation context for domain expertise← The dimension
2const domainExpertise = {
3  name: 'domain_expertise',
4  description: 'Depth of knowledge in primary work domain',
5
6  // State progression: ordered, mutually exclusive← The states
7  states: ['novice', 'familiar', 'competent', 'expert'],
8
9  // Behavioral signals that feed this context← The signals
10  signals: [
11    { type: 'vocabulary_complexity', weight: 0.30 },
12    { type: 'question_depth', weight: 0.25 },
13    { type: 'correction_patterns', weight: 0.25 },
14    { type: 'domain_term_usage', weight: 0.20 },
15  ],
16
17  // Confidence requirements per state← The thresholds
18  confidence: { novice: 0.6, familiar: 0.7, competent: 0.75, expert: 0.85 },
19};

Observation Contexts in Practice

Let me walk through a concrete example. Consider an AI product that helps engineers write documentation.

The product needs to make several personalization decisions for each user: what level of technical detail to include, what documentation structure to suggest, how much boilerplate to auto-generate, what tone to use, and how aggressively to suggest improvements.

Here are the observation contexts that inform those decisions.

Technical depth preference. States: overview-focused, balanced, implementation-detailed. Signals: how often users expand on generated summaries, whether they add code examples, depth of review comments they write.

Documentation style. States: minimal, structured, comprehensive. Signals: how much they modify generated structure, whether they add sections, how they handle tables and diagrams.

Domain vocabulary confidence. States: generic, domain-aware, domain-fluent. Signals: whether they correct domain terms, how they reference technical concepts, terminology consistency.

Collaboration orientation. States: solo-writer, reviewer-first, collaborative. Signals: sharing frequency, comment response patterns, co-editing behavior.

Improvement receptivity. States: resistant, selective, eager. Signals: suggestion acceptance rate, time spent on improvement recommendations, modification patterns after suggestions.

Five contexts. Each with clear states, defined signals, and confidence thresholds. Together, they give the product enough understanding to personalize meaningfully for each user without drowning in hundreds of unstructured attributes.

The State Progression Pattern

One of the most powerful aspects of observation contexts is state progression tracking over time. This is not just what state is the user in today - it is how have they moved through states and what does that trajectory tell us.

A user whose domain vocabulary confidence progresses from generic to domain-aware to domain-fluent over three months is learning and growing. The product should adapt - offering less basic explanation, introducing more advanced terminology, suggesting more sophisticated documentation patterns.

A user whose improvement receptivity moves from eager to selective to resistant over time is experiencing suggestion fatigue. The product should adapt differently - reducing suggestion frequency, focusing on higher-impact recommendations, giving the user more autonomy.

State progression creates a temporal dimension that flat attribute lists cannot capture. It turns observation contexts from snapshots into trajectories, and trajectories are where the deepest personalization insights live.

Common Design Mistakes

Having designed observation contexts across multiple products, I see the same mistakes repeatedly.

Too many contexts. Teams try to model everything and end up with 30+ contexts that never accumulate enough signal to be useful. Each context needs behavioral data to build confidence, and data is finite. Spread it across too many contexts and no single context reaches actionable confidence.

Overlapping states. States within a context should be mutually exclusive. If your states are beginner, intermediate, somewhat-advanced, and advanced, you have overlap that makes classification ambiguous. Clean states produce clean models.

Missing the product connection. Contexts must connect to product decisions. An observation context that accurately tracks a user attribute but does not inform any personalization decision is wasted infrastructure. Every context should have a clear answer to the question: what does the product do differently based on this context?

Static signals. The behavioral signals that feed a context should be reviewed and updated as the product evolves. New features produce new signals. Old features may become less relevant. The signal mapping is not set-and-forget - it is a living design.

Trade-offs

Context design requires deep product understanding. You cannot design good observation contexts without understanding what personalization decisions your product needs to make. This means the product team and the engineering team need to collaborate on context design - it is not a purely technical exercise.

Fewer contexts means faster confidence but coarser understanding. With 5 contexts, each context accumulates signal quickly and reaches actionable confidence in 10-15 interactions. But the personalization is relatively coarse-grained. With 15 contexts, personalization is more nuanced but each context takes longer to reach confidence. Finding the right balance depends on your usage frequency and interaction richness.

State progressions add complexity. Tracking progression over time is more valuable than tracking current state alone, but it requires temporal data storage, progression analysis, and trajectory interpretation. If your product does not need temporal personalization, current-state tracking is simpler and may be sufficient.

Observation contexts are product-specific. There is no universal set of contexts that works for every AI product. The contexts for a coding assistant are fundamentally different from the contexts for a health coaching app. Each product needs to design its own contexts based on its specific personalization needs.

What to Do Next

1. List every personalization decision your product makes or should make. Start from the product experience and work backward. When the product generates output, what varies per user? Length, depth, tone, domain focus, suggestion aggressiveness - each of these is a candidate for an observation context.

2. Map behavioral signals to those decisions. For each personalization decision, identify what observable user behaviors would tell you how to personalize. What does the user do that reveals their preference for concise versus detailed output? Those behaviors are your observation signals.

3. Design 8-12 observation contexts with clear state progressions. Use the framework in this article: dimension, states, confidence thresholds, and mapped signals. Clarity provides infrastructure for defining and tracking observation contexts as part of the self-model architecture. See how observation contexts work in practice.

Data without dimensions is noise. Observation contexts are the structure that turns interaction data into user understanding. Build the infrastructure for structured observation.

References

1. Twilio Segment’s 2024 State of Personalization Report
2. 2016 survey of 2,000 Americans by Reelgood and Learndipity Data Insights
3. Scientific American explains
4. cold start problem
5. Progress Software describes this core tension well

Related

• What We Learned Building Self-Models for Enterprise
• What Segment and Amplitude Miss About Your Users
• Three Layers of AI Personalization Architecture