97.5% of IT Questions Did Not Become Tickets: ShipStation Turned Slack History Into a Knowledge Base

ShipStation Global's IT team used to answer the same questions every day. The answers mostly already existed—in Confluence articles, on intranet pages, in some Slack thread from eight months ago. Finding the answer was the hard part.

They built a Slack bot called Deckhand, running on Amazon Bedrock AgentCore. In the 12 months before launch, more than 1,500 threads accumulated in the IT help channel, and the volume of questions was still rising. Employees are distributed across the United States, United Kingdom, Spain, Poland, Australia, and the Philippines, and more than one-fifth of questions occurred outside US working hours.

Three months after Deckhand launched: it processed 1,935 messages, started 1,221 conversations, and opened only 409 Jira tickets. It replied to 97.5% of channel messages and resolved 94% of after-hours questions. Positive feedback rate rose from 82% in April to 96% in June.

Opening, handling, and closing a ticket takes about 30 minutes. Those 812 conversations that did not become tickets saved about 406 hours over three months, nearly 1,600 hours a year. Employees got their time back, and the IT team no longer had to do repetitive intake.

The architecture is not complex; each component does one thing

Deckhand has four parts.

The Slack service uses FastAPI and Slack Bolt, runs on AWS App Runner. It receives Slack events, decides whether the bot should reply, and assembles context such as name, email, timezone, and manager from the user profile.

The agent service is written with the Strands Agents SDK and deployed in a container on the AgentCore runtime, backed by Claude on Amazon Bedrock. The inference loop is handled by the SDK; the model chooses tools, processes results, and iterates until it produces an answer.

Tools and knowledge live in a single Lambda function, exposed as MCP tools through AgentCore Gateway. It classifies requests, creates and updates Jira tickets, and analyzes images. The knowledge base uses Amazon Bedrock Knowledge Bases to index content from Confluence, the intranet, Slack, and marketing pages, stored in S3.

Operations and feedback use DynamoDB to store ratings, X-Ray for tracing, and CloudWatch for logs.

Employees send messages in the channel. The Slack service packages the full thread history and user context and calls the agent through the AgentCore runtime API. The agent queries the knowledge base and calls tools as needed. If it finds an answer, it returns it; if not, it asks a clarifying question or proposes opening a ticket.

Using RLM to mine Slack history is the most valuable part

Deckhand's most valuable data source is not the carefully maintained documents; it is the threads in Slack that have already been resolved. Employees describe a problem, and IT walks them through fixing it step by step. This history is an organic knowledge base, and the reviewer is the person whose problem was solved.

But it is dirty. The raw export is mixed with chit-chat, automated alerts, and unresolved threads. If an unverified fix or a messy summary is indexed, the knowledge base will give wrong answers with the same confidence. Extraction must be precise.

They used DSPy's Recursive Language Model (RLM). RLM does not stuff the entire Slack export into one prompt; instead, it puts it into a variable in a sandboxed Python REPL as a navigable memory space. The model iterates over it, writing code to retrieve and classify each thread as resolved, unresolved, or escalated, and filtering out announcements and non-questions.

The standard for determining resolved is strict: the complete, actionable fix must appear in the thread itself.

For each resolved thread, RLM then orchestrates recursive sub-LLM calls to extract structured fields, with each field validated by a Pydantic model. Fields include problem summary, original question, solution, whether self-service or ticket required, self-service steps, affected applications and systems, error messages, and tags. If a sub-LLM returns something that does not match the schema, RLM's code loop catches the validation error in the interpreter and retries. Results are organized into markdown documents by problem and uploaded to S3 for the knowledge base to index.

The reason for choosing RLM is that it can keep the entire Slack history in a persistent sandbox variable space, reason iteratively, and produce consistent structure. When an employee encounters a problem a colleague has solved before, Deckhand provides a verified solution, usually as numbered steps.

Knowing when not to speak

When an AI bot is placed in a shared channel, social failure comes before technical failure. The logic Deckhand polished the most is deciding when to shut up.

The first judgment it makes on every message is: is this an announcement, a social message, or a conversation between two people? If so, it outputs SKIP, and the Slack service does not reply. In a thread, it only replies to the original poster. If the last reply came from another human helper, it stays silent because a human has taken over. If a reply starts by @-mentioning someone, that message is not for the bot.

Security and validation do not rely on prompts

Prompt rules can change behavior, but they cannot replace access control. Deckhand's enforcement is in code and infrastructure.

Creating a ticket requires explicit confirmation. When the bot proposes opening a ticket, the Slack service adds a Create Ticket button; only when it is clicked is a confirmation sent through the agent. Ticket updates only accept valid keys matching the helpdesk project pattern and reject empty comments. Request classification runs as a separate Amazon Bedrock Converse API call, returning structured fields that are first validated with Pydantic and then sent to Jira. The image tool can only download from Slack's file host, authenticated with the bot token, with at most five images at a time and each no larger than 5 MB, and sends them to the vision model through the Converse API. Slack tokens, signing secrets, and Jira credentials are stored in AWS Secrets Manager, and each service and tool runs under an IAM role scoped to its own permissions. The prompt also includes behavioral guardrails: do not request passwords or MFA codes, do not write credentials in ticket comments, and do not expose profile metadata used for ticket fields.

Results

Deckhand launched in April 2026 and grew from 30 messages per day to 100 by early summer. In three months it processed 1,935 messages, started 1,221 agent conversations, and opened 409 Jira tickets. One-third of questions came in after hours, and Deckhand answered 94% of them. People who would otherwise have had to wait until the next morning got answers that same evening.

Chris Collins, Senior Director of IT at ShipStation, said in the AWS blog that they built Deckhand because getting help should be as simple as asking a question in Slack, rather than interpreting a ticket form or sending email into the void. They believe tickets represent operational friction, not productivity.

What is interesting about this case is not how new the frameworks are. AgentCore runtime hosts the agent, Gateway turns Lambda into MCP tools, and Knowledge Bases handles retrieval; the team built no servers and maintained no orchestration code. What is truly counterintuitive is the choice of data source: the most valuable knowledge is not in carefully maintained wikis, but in Slack history that no one has organized and that is mixed with chit-chat and alerts. What RLM does is turn that dirty data into verifiable answers.

Reference source: AWS Machine Learning Blog, the original article co-written by the ShipStation Global team and AWS Solutions Architects.