Agents Inherit Your Permissions—and the Folder You Shared by Mistake in 2019

Box CEO Aaron Levie posted a tweet announcing a partnership with NVIDIA to build “advanced context retrieval” for AI agents in enterprise settings. The approach combines lexical search and semantic search inside Box so agents can find the most relevant information, while the entire process complies with enterprise security policies, access controls, and data governance rules—agents can only touch data they are authorized to touch.

His exact words were: Files 🤝 GPUs.

100x Agents, 90% Unstructured Data

Levie’s claim sounds a bit bold: the number of agents in enterprises that interact with data and systems will be 100 times the number of employees at companies today.

The success or failure of every agent deployment depends on whether it can reliably get the critical context it needs to make decisions, do research, and execute tasks. Yet 90% of enterprise data is unstructured, and this context is scattered across documents, contracts, marketing assets, research materials, financial plans, and design files. So the problem isn’t whether the model is smart enough; it’s whether the model can read that pile of files.

The Comments Aren’t Discussing Retrieval

Within minutes of the tweet being posted, almost all the replies veered away from retrieval itself.

One user, Elliott Michael Smith, offered a very simple acceptance test: “I’ll judge this search by whether it finds the signed contract or an old draft.”

Will asked specifically: Does an agent have its own set of Box permissions, or does every retrieval directly inherit the ACL of the user who initiated the request?

Somi’s answer had a bit of dry humor: An agent inherits your Box permissions—and, by extension, every folder that someone accidentally over-shared back in 2019.

Prasen Shelar took it a step further: before connecting Box, you’d better go through all those old publicly shared external-link folders, because the agent will dredge up files no one remembers sharing.

Ammar Haider pointed out the ambiguity hidden in the phrase “only access permitted data”: What does permitted mean—everything the user can see, or only the subset this task actually needs? He said today’s default is to let the agent inherit all of the user’s permissions, which is far broader than what most tasks actually require.

Dominykas Bickus’s complaint was closer to everyday life: 100x agents, and you still have to wait for a SharePoint approval.

Another point many people overlooked: Jaydeep Makwana said context is only half the battle; the other half is what agents are allowed to write after they get the context. Most architecture diagrams skip the permissions box.

The Same Problem in 1990

John Brewton dug up some history. Howard Shao and John Newton founded Documentum in June 1990, and the earliest customer requirements were organizing Boeing 777 training manuals and Syntex’s FDA filings. His take: an agent that needs Box and NVIDIA to find the right file is still solving the same old problem.

That comparison is worth pausing on. Document management as a category has existed for more than thirty years, and the core difficulty has never been storage—it’s locating things and permissions. Now it has a new name, context retrieval, but what has to be handled underneath hasn’t changed much.

Access Became the Product

One reply contained only five words: access is the product now.

Another user, zrqgood, put it more fully: once the number of agents grows 100x, the bottleneck will shift from model capability to permission governance. Every agent needs an independent identity, least privilege, and auditable access logs; otherwise, the extra 100x becomes 100x the data-leak surface. Whichever enterprise content layer first makes this the default configuration will own the entry point for agents.

Retrieval is responsible for finding files; governance is responsible for deciding who can see and who can write. The former is the selling point of this partnership; the latter is the gate that determines whether a deployment can actually go live. Box and NVIDIA have built lexical-plus-semantic hybrid retrieval into the product this time, and security policies and access controls are written into the announcement. But the details the comments keep asking about—whether agents have independent identities, whose ACL is inherited during retrieval, and how to clean up historical over-sharing—are not answered in the announcement.

Akshay Saini added one last point that is easy to skip: an agent can find the final design file, but it still doesn’t know why other options were rejected—unless someone wrote down the reasons at the time.

On this one, GPUs can’t help.