Technical Research: The Memfit AI Built-in Knowledge Base System and Agentic RAG Implementation

When an AI Agent runs into something it does not know

In the previous article: Memfit AI: A Production-Grade AI Agent That Does Not Get Lost Over N Hours of Continuous Penetration Testing, we talked about how Memfit AI works autonomously like a real human penetration testing engineer — planning tasks, executing attacks, and dynamically adjusting strategy. It ran for more than two hours and produced 18 vulnerability findings and a full penetration testing report.

But there is one problem we sidestepped back then.

What if, during execution, the AI Agent encounters an enterprise-internal security specification it has never "seen" before — for example GB/T 34944-2017, the Java Source Code Vulnerability Testing Specification? This national standard defines the vulnerability types, testing methods, and acceptance criteria that should be considered in a Java security audit. Large model training data does not necessarily contain these details, and even if it does, the version and accuracy cannot be guaranteed.

This scenario is very common in real work. Security teams have their own checklists, enterprises have private compliance requirements, and a client's security baseline documents are almost never published. If the AI Agent can only rely on the large model's own "memory," its ability to handle this kind of private knowledge is zero — or even worse, it may "hallucinate" answers that look reasonable but are completely wrong.

So how do we solve this? The common solution on the market is to bolt on a RAG system or connect to an external knowledge base service through the MCP protocol. This certainly works, but there is a fundamental split: the knowledge base is the knowledge base, the Agent is the Agent, the two exchange data through a pipe, and the Agent is very passive in how it uses the knowledge — query once, fetch once, use whatever comes back.

The Memfit AI knowledge base system takes a different path. We made the knowledge base an "internal organ" of the Agent. The whole system runs fully locally, and the Agent can query and explore the knowledge base autonomously and across multiple rounds during task execution. This means that after getting the first round of search results, the Agent judges whether the information is sufficient based on those results; if not, it adjusts its query strategy and searches again — iterating up to 3 to 5 rounds until it has gathered enough information to make a decision.

I believe this is the most noteworthy design of the Memfit knowledge base. The relationship between the knowledge base and the Agent should be one of "unity of knowledge and action" — the knowledge base is responsible for remembering massive amounts of private data and specialized content that there is no time to train into the model or that cannot be trained into the model at all; the Agent makes judgments and takes action based on this knowledge. Only when the two are fused into one system can they truly deliver value.

In this article, we will give you a complete walkthrough of the Memfit knowledge base system:

01 How to create and use a knowledge base

02 The difference between the two knowledge construction modes and how to choose

03 Query validation and the actual effect of Agentic RAG

04 How the Agent invokes the knowledge base in a production task

05 The underlying technical implementation — the complete chain from index construction to multi-round search

Entering the knowledge base

The Memfit AI knowledge base feature is integrated into the main interface. After opening Memfit, you can see the "Knowledge Base" tab in the top navigation bar; click it to enter the knowledge base management interface.

On the left side of this interface, we can see the list of existing knowledge bases — including both online and local knowledge bases. The area on the right is the AI Agent's conversation window; the knowledge base and the Agent share the same workspace. This layout itself hints at the close relationship between the two.

Creating a knowledge base

Now let's do it hands-on. Click the "Quick Create Knowledge Base" button, or find the "New Knowledge Base" entry below the knowledge base list.

The system pops up a parameter entry window:

We need to fill in several key parameters:

01 Knowledge base name: Give the knowledge base a semantic name, so it is easy to reference later inside the Agent.

02 Upload file: Supports direct drag-and-drop or selecting a file path. You can import PDFs, documents, text, and other formats.

03 Build mode: There are two options here — "Enhanced Knowledge Graph Index" and "Build Knowledge Index Only."

04 Tags: Optional category tags, used for organizing and filtering.

The two build modes

The choice of build mode directly affects the storage structure and query capability of the knowledge base. We use a table to compare them:

The "Enhanced Knowledge Graph Index" mode performs deep analysis on the document: first it slices the content, then it extracts entities and relationships through ERM (Entity Relationship Model) to build a knowledge graph; at the same time it generates a question index for each slice, used for vector retrieval. The whole process involves multiple rounds of AI calls, so construction takes longer, but query quality is also higher.

The "Build Knowledge Index Only" mode is lighter. It skips the entity-relationship extraction step and goes straight to generating a query-enhanced index from the slice content, then vectorizes it and stores it. If your document is already high-quality structured knowledge (such as API documentation or technical specifications), this mode offers better cost-effectiveness.

Hands-on: importing GB/T 34944-2017

Let's walk through the entire flow with a concrete example. This time we choose to import GB/T 34944-2017, the Java Source Code Vulnerability Testing Specification national-standard document.

In the popup, fill in the knowledge base name as "My Knowledge Base," set the upload file to the local PDF file path, and choose "Enhanced Knowledge Graph Index" as the build mode — because the national-standard document contains a large number of associative relationships among vulnerability classifications, testing methods, and acceptance criteria, which is well suited to building a knowledge graph.

After clicking "OK," the build process begins. You can watch the build progress in real time on the execution panel:

Several key stages can be observed in the log:

01 Document parsing: The system identifies the PDF file type and parses the document content.

02 Slicing: The long document is split into multiple chunks, with each chunk preserving semantic completeness.

03 Entity graph construction: "start build entity graph concurrency" indicates that the system starts extracting entities and relationships in parallel.

04 Indexing: Once extraction is complete, entities, relationships, and vector indexes are written into local storage.

After the build completes, we can see the result in the knowledge base interface.

Knowledge graph visualization

This is the most intuitive part. Switching to the graph view of the knowledge base, you can see the network of entities and relationships automatically extracted by the system:

Select any node to view that entity's detailed attributes and associated relationships. This graph will play an important role in subsequent queries — when the Agent searches for an entity, it can use K-hop traversal to find other knowledge points connected to it, which is the basis for multi-hop knowledge queries.

Query validation

The knowledge base is built. Now let's validate the query effect.

Simple recall

In the "AI Recall" functional area of the knowledge base interface, we can directly test the knowledge base's recall capability. Click the "AI Recall" tab and enter a question in the search box, for example:

How should the security settings related to logging in Java be handled?

The system converts the question into a search query and retrieves relevant content from the selected knowledge base. The @ selector below the search box can scope the search — whether to search across all knowledge bases or only within one specific knowledge base.

Agentic RAG recall

This is where it gets really interesting.

When we issue an Agentic RAG query, the system turns the user's question into an Agent search task. The AI autonomously decides the search strategy and then progressively explores the content in the knowledge base.

We can observe several key behaviors:

01 The AI first understands the intent of the user's question, splitting or generalizing it into multiple sub-queries.

02 It calls search functions such as search_knowledge_semantic and search_query.

03 It evaluates each round of search results to judge whether the information is sufficient.

04 If more detail is needed, it automatically adjusts the query content for another round of search.

Query result

Let's look at a typical query result. We asked a question about "Java logging security settings," and the system returned a structured answer:

There are several things worth noting about this result. The AI performed deduplication, merging, and structured organization on multiple retrieved knowledge fragments, forming inductive headings such as "Logging Security Guidelines," with specific content organized along dimensions like "Log access control" and "Do not log sensitive information." Every point can be traced back to an original knowledge base entry — this content comes from multiple sections of GB/T 34944 related to log security, and the AI has correlated them.

Using the knowledge base directly from the Agent

So far, what we have shown is the scenario of manually querying inside the knowledge base interface. So here comes the more core question: when the Agent executes a task, how does it use the knowledge base?

In the Memfit AI conversation interface, we can reference a knowledge base through the @ symbol.

During task planning and execution, the Agent autonomously decides whether to query the knowledge base, which knowledge base to query, and what question to use — based on the needs of the current task.

Take a concrete example. We let the Agent answer a composite question that touches on multiple areas of Java security. During execution, the Agent automatically triggered an Agentic search of the knowledge base.

We search a composite question and can see the two-round Agentic search process. The first round retrieved partial results; after evaluation the Agent concluded that some dimensions of information were still missing, so it adjusted the query content and ran a second round of search.

This kind of self-adjustment capability is something that a typical standalone knowledge base system cannot do — they only passively receive queries and return results. The Agent's "proactive exploration" is the real value of Agentic RAG.

This is also why we say "unity of knowledge and action." The knowledge base provides memory, the Agent provides the power to act, and the two work closely together within the same runtime. When the Agent encounters an uncertain question, it acts like an experienced engineer flipping through a reference manual — searching repeatedly, cross-checking, until it has enough confidence to make a judgment.

How does Memfit implement all of this?

Having covered the user experience, let's look at the underlying technical implementation. This part is aimed at readers interested in RAG systems and Agent architecture; we will analyze several key modules together with the actual code.

Enhanced index construction and knowledge graph construction

From upload to queryable, a document must go through a complete processing pipeline. We use a flowchart to illustrate:

The difference between the two modes is reflected in the middle stage. The "Enhanced Knowledge Graph Index" mode walks the full chain: ERM extraction + question index + vectorization; the "Build Knowledge Index Only" mode skips ERM and goes directly from slice content to generating the question index.

Question index construction is a step worth expanding on. The system calls the AI to generate a set of "retrieval questions" for each document slice — these questions simulate what a real user might search for, covering multiple dimensions such as how-to, concept definition, root-cause analysis, best practices, and troubleshooting. The core logic lives in the BuildIndexQuestions function:

funcBuildIndexQuestions(rawInput []string, aiService aicommon.AICallbackType)
(map[string][]string, error) {
    linedInput := utils.PrefixLinesWithLineNumbers(rawInput)
    query, err := LiteForgeQueryFromChunk(indexBuildPrompt, "",
chunkmaker.NewBufferChunk([]byte(linedInput)), 200)
    // ...
    result, err := aicommon.InvokeLiteForge(query, forgeOpts...)
    // ...
    entries, err := index2KnowledgeEntity(result.Action, rawInput)
    return entries, nil
}

This function does several things: it adds line-number markers to the raw text, assembles a prompt with detailed instructions, invokes LiteForge (Memfit's lightweight AI execution engine) to generate a list of questions, and finally maps the questions to the corresponding answer fragments in the original text. Each generated question carries a precise line-number range pointing to the specific paragraph in the original text that answers it.

These questions are then vectorized and, together with the original-text vectors, stored in an HNSW (Hierarchical Navigable Small World) vector index. At query time, the user's question is matched against both the "original content vectors" and the "generated question vectors" — because the generated questions are closer to user search habits, recall improves significantly.

Knowledge graph construction is completed through ERM entity-relationship extraction. The system uses the AI to identify entities from the document (such as vulnerability types, security standards, testing methods) and the relationships among them (such as "belongs to," "testing method," "mitigation"), and builds them into a directed graph. This graph is stored in a local database and supports K-hop multi-hop traversal queries.

Multi-strategy search pipeline

After a user query comes in, the system does not just do a single simple vector search. In fact, Memfit's search pipeline runs multiple search strategies in parallel and then fuses and ranks all results.

The five search strategies each have a different focus, corresponding to the SearchHandler interface in the code:

01 Basic vector search (Basic): Uses the original query to do direct vector similarity matching. This is the most basic semantic search.

02 HyDE (Hypothetical Document Embedding): The AI first generates a "hypothetical ideal answer" for the user's question, then uses this answer as the query content to search. Because the "answer" and "document content" are closer in semantic space, it often finds results that a direct question search misses.

03 Split Query: Breaks a complex question into multiple independently retrievable sub-questions, searches them separately, and merges the results. For example, "Java logging security best practices and common vulnerabilities" would be split into two sub-questions that are searched separately.

04 Generalize Query: Lifts a concrete question to a higher conceptual dimension for search. For example, "Redis unauthorized-access vulnerability" would be lifted to a search around "NoSQL database security risks," capturing more macro-level survey documents.

05 Exact Keyword Search: Extracts core keywords from the question and does precise term matching, compensating for vector search's weakness on proper nouns.

Let's take HyDE as an example and look at the concrete implementation. The core logic of the HypotheticalAnswer function is to have the AI generate an "ideal answer summary" kept within 100 characters:

func(h *LiteForgeSearchHandler) HypotheticalAnswer(ctx context.Context, query string)
(string, error) {
    prompt := `你是⼀个精通信息检索的AI助⼿。
根据⽤户提出的【问题】，精准地提炼出其核⼼概念，
并⽣成⼀段信息密度极⾼的"理想答案摘要"。
严格控制在100字以内。`
    // ...
    result, err := aicommon.InvokeLiteForge(inputPrompt, ...)
    document_paragraph := result.GetString("hypothetical_answer")
    return document_paragraph, nil
}

This hypothetical answer is vectorized, and its vector is then used to search real documents — this trick comes from the academic HyDE method and is very effective at improving recall in practice.

AI reranking and refinement

After the five strategies run in parallel, a large number of candidate results are produced, inevitably with duplication and noise. Two "refinement" steps are then applied:

First pass: RRF reranking (Reciprocal Rank Fusion)

Each search strategy sorts its own result set by relevance. The RRF algorithm fuses ranking information from different strategies into a unified score. The core formula is:

$$RRF(d) = \sum_{i=1}^{n} \frac{1}{k + rank_i(d)}$$

Where $k$ is a smoothing constant (Memfit defaults it to 60), and $rank_i(d)$ is the rank of document $d$ under the $i$-th strategy. If a document ranks near the top across multiple strategies, its RRF score will be high. The implementation is very concise:

funcRRFRank[TRRFScoredData](scoredDataList []T, k int) []T {
    // 按搜索⽅法分组，计算每个⽂档在各⽅法中的排名
    // 然后对每个⽂档求 RRF 累加分
    // 最后按 RRF 分数降序排列
}

The benefit of RRF is that it does not depend on the absolute value of the original scores — it only looks at rank — so it can safely fuse results from different scoring systems.

Second pass: AI summary refinement

After RRF ranking, the Top-K results are taken. If EnableAISummary is on, the system sends these results to the AI in aggregate to generate a refined answer. The AI's work instruction is very clear:

Answer the user's question using only the content of the knowledge base entries. Do not introduce external knowledge or subjective inference; if no answer can be derived from the entries, directly reply "No relevant information was found in the knowledge base."

This ensures that the final answer is strictly based on knowledge base content and that the large model's "hallucinations" do not leak in.

Agentic self-adjustment and multi-round iteration

At this point, we have explained the complete flow of a single query. So how is the "multi-round search" of Agentic RAG actually implemented?

The core mechanism can be broken into three steps:

01 Search plan formulation.

After receiving a question that needs knowledge support, the Agent first analyzes which knowledge dimensions the question involves. For example, "the complete workflow of a Java security audit" may involve vulnerability classification, testing methods, tool usage, reporting standards, and more. The Agent uses this to decide the query content for the first round of search.

02 Result evaluation and adjustment.

After each round of search completes, the Agent evaluates whether the results so far are sufficient to answer the user's question. If it finds that some dimension's information is missing — for example, it found vulnerability classifications and testing methods but nothing about reporting standards — the Agent adjusts the next round's query to specifically fill that gap.

03 Multi-round result merging.

After all search rounds complete (or once the Agent judges the information to be sufficient), the system deduplicates the multi-round results (the same knowledge fragment may be hit in multiple rounds), merges them (high-relevance fragments are aggregated together), and finally generates a structured answer.

This process is very similar to how a real person consults material: skim through once first, notice that one aspect is thin, then do targeted deep digging. The difference is that the Agent can complete this iteration in a few seconds, whereas a human might spend half a day flipping through documents.

Of course, there is an engineering trade-off here. The more search rounds, the more comprehensive the recalled information, but the longer it takes. We have temporarily set an upper limit of 3 to 5 rounds; in most scenarios this range is enough to cover the various dimensions of a composite question. We will continue to tune this parameter based on real usage data.

Can the Agent also consult material while doing work?

Earlier we demonstrated querying inside the knowledge base interface and @-mentioning the knowledge base in conversation. But both of these stay at the "Q&A" level — the user proactively asks, the system passively answers.

So what about a real production scenario?

If we give the Agent a concrete task — such as "perform a code audit on a Java SpringBoot2 project" — can the Agent go look up the security specifications in the knowledge base on its own during execution?

Let's try it. In the Memfit AI dialog, enter the task instruction, and at the same time attach the knowledge base we just created through the @ symbol:

Note this operation: we tell the Agent to "perform a code audit," and specify that it should use the content of @MyKnowledgeBase to assist the audit process. After receiving the instruction, the Agent carries the knowledge base as its own "reference material" through the entire task execution chain.

Consult knowledge first, then plan the task

After the Agent starts the task, the very first thing it does is interesting — it does not directly start scanning code. It first goes to the knowledge base and looks up material for one round.

From the execution log, we can see that the Agent triggered a knowledge base search already during the planning phase. It retrieved the vulnerability types and testing methods that need attention in a Java code audit from GB/T 34944, and then formulated the task plan based on this knowledge.

This order is worth noting: consult material first, then settle on the plan. This is exactly the behavior pattern of an experienced security engineer receiving an audit task — after taking on the project, first flip through the security specifications, figure out what to audit and by what standard, and only then get hands-on.

Even more worth noting is that during execution we saw the dynamic task planning capability introduced in the previous article. After consulting the knowledge base, the Agent, based on the retrieved security specification entries, proactively modified its own task list — breaking the originally rather generic "code audit" into more specific sub-tasks, such as detection for SQL injection, detection for XSS, detection for logging security, and so on. The goal and method of each sub-tasks aligned with the specification requirements in the knowledge base.

This is the chemical reaction produced when the knowledge base and the Agent are deeply fused: the content of the knowledge base directly influences the Agent's behavioral decisions, and the Agent's task planning becomes more precise and more targeted as knowledge is injected.

Execution result

Of course, not surprisingly, we can harvest some vulnerabilities:

When we wait for the system to finish executing, we can see the following content:

This flow is fundamentally different from the earlier "@ the knowledge base in conversation to ask a question" scenario. A query in conversation is one-shot — once the Agent asks, it is done. During task execution, however, the knowledge base is invoked repeatedly, running through the whole process of planning, execution, judgment, and reporting.

The Agent genuinely treats the knowledge base as a "reference manual" it carries with it — flipping it open when needed, setting it aside when not, but always keeping it on hand.

Recap

We accomplished four things in this article:

First, we walked the entire flow at the operational level.

From entering the knowledge base interface, creating a knowledge base, choosing a build mode, and importing a PDF document, to viewing the knowledge graph and running query validation — all of these steps can be completed within a few minutes in Memfit AI.

Second, we showed the actual effect of Agentic RAG.

Through the query test on GB/T 34944, we saw how the Agent autonomously searches across multiple rounds and progressively refines its answer. This capability is especially obvious when handling composite questions — information that a single round of search cannot retrieve is often covered after multi-round iteration.

Third, we validated the collaboration between the knowledge base and the Agent on a real production task.

The execution of the code audit task shows that the Agent proactively consults the knowledge base during the planning phase, adjusts the task plan based on the retrieved security specifications, and continuously references knowledge base content throughout the execution chain to assist judgment. The knowledge base's influence on the Agent runs through the entire process from planning to output.

Fourth, we went deep into the underlying implementation.

From question index construction to five-strategy parallel search, from RRF reranking to AI summary refinement, to Agentic multi-round iteration — the entire chain has one consistent design goal: make the knowledge base an organic part of the Agent, and turn "looking up knowledge" into a proactive, iterable, strategy-driven process.

The Memfit AI knowledge base system is now live in the product. If you are interested in trying it out, you can visit https://memfit.ai/ to download Memfit AI. During the closed-beta promotion phase, no token fees are charged. You can of course also configure your own AI API key to use it.

memfit:: A knowledge base that remembers, visible power to act

In upcoming articles, we will introduce more advanced capabilities of Memfit AI — including customization of the Skills system, multi-Agent collaboration, and how to apply the knowledge base to real security audit workflows.

---

This article was first published on the Yak Project official account. Read the original (Chinese).