Your AI Agent Is Logged In. That Doesn’t Mean It Should Have Access.

An AI memory app starts off cute.

I ran into this while building RecallMEM, a persistent AI chatbot with a memory framework underneath it. The first version only had my own notes, so the permission model was basically “it’s mine.” That worked fine until I started designing for memory that comes from real data from Gmail, Google Calendar, Slack, shared docs, and team notes.

That’s when retrieval stopped being the hard part. A vector database can find relevant memories, but it can’t tell you whether the agent should be allowed to use them.

I personally spent a lot of time making retrieval deterministic with Postgres, pgvector, TypeScript validators, and a simple rule: the LLM never touches the database. That helped, but it also made the next problem obvious. Even perfect retrieval isn’t authorization.

Relevance is not permission.

This Code Looks Safer Than It Is

Here’s the shape a lot of early memory code ends up in:

async function getAgentContext(userId: string, question: string) {
  const embedding = await embed(question);
  return db.memories.findSimilar({ userId, embedding, limit: 5 });
}

This looks reasonable, and that’s what makes it dangerous.

There’s a userId. We’re not searching everyone’s memories. We embed the question, pull the nearest facts, and hand them to the model.

For a single-user prototype, this is usually fine. The boundary is simple because the data is yours.

But for anything multi-user or connected to real tools, it’s just not enough.

A memory might have come from a private note, a Gmail thread, a Slack channel, a shared document, or a customer record. Some of those sources are personal. Some belong to a team. Some are visible in one workspace but not another.

The agent might be allowed to read a memory but not write one. It might be allowed to draft an email but not send it. It might be allowed to read calendar events but not change them.

A vector database can realistically only answer one question: what’s similar? Authorization answers another: what’s allowed?

Those aren’t the same question, and mixing them up is how a memory feature quietly becomes an access control problem.

Login Is The Easy Part

Most developers meet identity through login.

You add a sign-in button, get a session, read a user profile, protect a route, and store the user ID. That’s real work, but it’s not the whole job.

Authentication tells you who the user is. Authorization tells you what that user, or agent, can actually do. Delegated access is the narrower question: what can this agent do on the user’s behalf?

The thing is, agents don’t just render screens. They make decisions inside loops. They pick tools. They retrieve context. They chain actions. Sometimes they act before the user has seen exactly what they’re about to do.

That’s why a signed-in user is not the same as an authorized agent. Authentication checks ID at the door. Authorization decides which rooms they’re allowed to enter and for how long.

Give Agents Fewer Keys

A better design treats every tool and data source as its own permission boundary.

Not in a vague “be careful with data” way. In code.

const agentPermissions = {
  memory: ["read", "write"],
  gmail: ["read"],
  calendar: ["read", "write"],
  slack: [],
};

This agent can read and write memory. It can read Gmail, but it can’t send. It can read and write calendar events, and it has no Slack access at all.

I think that distinction matters. Gmail read isn’t Gmail send. Calendar read isn’t calendar write. Memory search isn’t memory delete. Drafting isn’t sending.

For anything sensitive, the agent should pause. Sending an email, deleting memory, accessing a new data source, or writing to a shared workspace should require explicit approval. The agent can prepare the action. The user should approve it.

I don’t want Gmail, Calendar, or Slack tokens anywhere near prompts, logs, or memory records. The agent should ask for access through a controlled layer, not carry the token around itself. That’s where something like Auth0 Token Vault fits.

The bigger point: agent access needs boundaries you can inspect, enforce, and revoke.

Roles Get Weird Fast

RBAC is fine until the resource you’re protecting is one remembered fact inside one agent response.

“Admin” and “user” don’t answer the questions agentic apps actually ask. Can this user read this specific memory? Can this agent use this document in its response? Can this agent read from Gmail but not send through it?

AI memory makes authorization smaller and stranger. You’re not just protecting pages anymore. You’re protecting facts, messages, documents, embeddings, tool calls, and actions.

That’s where relationship-based authorization starts to really make sense. The model becomes relational fast: a user owns this memory, belongs to that team, can read documents in this workspace, and delegated this agent access to one source but not another.

Broad roles stop being enough here. That’s exactly the kind of relationship problem Auth0 FGA is built to model.

Then retrieval can work with authorization instead of pretending to replace it.

async function getAuthorizedAgentContext(
  user: User,
  agent: Agent,
  question: string
) {
  const embedding = await embed(question);

  const candidates = await db.memories.findSimilar({
    embedding,
    limit: 20,
  });

  const allowed = [];

  for (const memory of candidates) {
    const ok = await canAccess({
      user,
      agent,
      action: "read",
      resource: memory,
    });

    if (ok) allowed.push(memory);
    if (allowed.length === 5) break;
  }

  return allowed;
}

The vector database gives candidates, and the authorization layer decides which ones are actually allowed.

In a real system, you should narrow the search space before retrieval when you can, then enforce authorization again before anything reaches the model. Don’t let the model see data and then ask it to decide whether the user should have seen it. By then the boundary has already failed.

The model can help decide what’s useful. It shouldn’t decide what’s allowed.

Better Boundaries Make Better Agents

Agents get useful when they can touch real things. That’s also when they get risky.

An agent that can’t access anything is safe but useless. An agent that can access everything is useful right up until it isn’t.

Logging in identifies the user. It doesn’t authorize every agent action. Scoped delegated access is safer than blanket permissions. Vector search shouldn’t be your access control system.

This doesn’t mean every prototype needs a complex identity model on day one. Prototypes are allowed to be prototypes. But once an agent connects to real user data, identity stops being just login plumbing and starts shaping how the product behaves.

Not just login or sessions. Not just “does this user exist?”

The real question is: what is this agent allowed to do next?

Better models won’t fix bad boundaries.

Chris Dabatos

DevRel Engineer, Builder, and Technical Storyteller based in Las Vegas. He builds things with AI and writes about what breaks.