Generate Documentation from Code Docstrings
AST-based reader that extracts only docstrings from a code directory into LlamaIndex Documents, avoiding the token cost of feeding full source to an LLM.
Maintainer of this project? Claim this page to edit the listing.
0.14.22Add to Favorites
Why it matters
Transform your codebase's docstrings into structured documents for LLM analysis. Generate comprehensive documentation or build intelligent code-buddy chatbots without tokenizing your entire codebase.
Outcomes
What it gets done
Parse docstrings from Python modules, classes, and functions.
Convert extracted docstrings into LlamaIndex Documents.
Create a knowledge base for LLM-powered documentation generation.
Build a code-buddy chatbot for your codebase.
Install
Add it to your toolbox
Run in your project directory:
curl -fsSL https://spark.entire.vc/get/li-reader-readers-docstring-walker | bash Overview
Intro
An AST-based LlamaIndex reader that walks a Python code directory and extracts only docstrings from modules, classes, and functions into Documents, avoiding the token cost of feeding full source code to an LLM. Use for documentation generation or code-explainer chatbots on well-documented codebases. Not suitable for sparsely documented code or when actual code logic, not just its documented interface, needs to be understood.
What it does
DocstringWalker addresses a middle ground between two common but costly approaches to generating documentation or code-buddy chatbots from a codebase: reading rich docstrings by hand, or feeding an LLM the full source of every file (which burns tokens, time, and compute). Many open-source libraries such as Scikit-learn or PyTorch carry docstrings rich enough to contain LaTeX equations or detailed usage examples, and DocstringWalker is built to extract exactly that content, and nothing else.
It works by recursively walking a local code directory using Python's built-in ast module - never executing or otherwise ingesting the code itself - and parsing docstrings from modules, classes, and functions as it goes. It builds a dependency graph between the extracted docstrings (tracking which classes and functions belong to which module), then converts the result into LlamaIndex Document objects that can be fed into a VectorStoreIndex and queried like any other LlamaIndex data source.
The DocstringWalker class takes two configuration parameters: whether to ignore __init__.py files (some projects leave them empty, others put meaningful package-level documentation in them, so this is left as a choice) and whether to fail on error (since Python's ast module raises a SyntaxError on malformed files, this flag controls whether that aborts the whole walk or is silently skipped).
The documented example walks the reader across its own source directory as a self-test, showing the resulting Document text (module name, docstring, nested class and function docstrings) and then indexing and querying that output with a natural-language question about the module's main functions. The query engine's response in the example correctly summarizes several of the module's own documented functions, including load_data, process_directory, read_module_text, and parse_module, each described by what it returns (a Document, a tuple of documents and a networkx dependency graph, or module text) - demonstrating that the docstring-only extraction preserves enough structure for accurate Q&A without ever passing the underlying code to the LLM.
When to use - and when NOT to
Use this reader when you want to generate documentation or build a code-explainer chatbot from a well-documented codebase and want to minimize LLM token spend by feeding in only docstrings rather than full source. It's a good fit for libraries or projects where docstrings are detailed and reasonably complete. It is not appropriate when a codebase has sparse or missing docstrings, since there would be little to extract, or when the actual code logic (not just its documented interface) needs to be understood or queried - in that case a full-source reader is the correct tool instead.
Inputs and outputs
Input is a local directory path containing Python source code. Output is a list of LlamaIndex Document objects, one per module, each containing the extracted docstring text for that module along with its classes and functions, plus an internal networkx graph capturing the dependency structure between them.
Integrations
Built on Python's standard ast module for parsing and networkx for the dependency graph, producing output as native LlamaIndex Document objects ready to feed into a VectorStoreIndex and query via a standard LlamaIndex query engine.
Who it's for
Developers building documentation generators or code-buddy chatbots for well-documented Python codebases who want to keep LLM token usage low by indexing only docstrings instead of full source files.
Source README
Intro
Very often you have a large code base, with a rich docstrings and comments, that you would like to use to produce documentation. In fact, many open-source libraries like Scikit-learn or PyTorch have docstring so rich, that they contain LaTeX equations, or detailed examples.
At the same time, sometimes LLMs are used to read the full code from a repository, which can cost you many tokens, time and computational power.
DocstringWalker tries to find a sweet spot between these two approaches. You can use it to:
- Parse all docstrings from modules, classes, and functions in your local code directory.
- Convert them do LlamaIndex Documents.
- Feed into LLM of your choice to produce a code-buddy chatbot or generate documentation.
DocstringWalker utilizes only AST module, to process the code.
With this tool, you can analyze only docstrings from the code, without the need to use tokens for the code itself.
Usage
Simply create a DocstringWalker and point it to the directory with the code. The class takes the following parameters:
- Ignore init.py files - should init.py files be skipped? In some projects, they are not used at all, while in others they contain valuable info.
- Fail on error - AST will throw SyntaxError when parsing a malformed file. Should this raise an exception for the whole process, or be ignored?
Examples
Below you can find examples of using DocstringWalker.
Example 1 - check Docstring Walker itself
Let's start by using it.... on itself :) We will see what information gets extracted from the module.
#### Step 1 - create docstring walker
walker = DocstringWalker()
#### Step 2 - provide a path to... this directory :)
example1_docs = walker.load_data(docstring_walker_dir)
#### Let's check docs content
print(example1_docs)
"""
[Document(id_=..., embedding=None, metadata={}, excluded_embed_metadata_keys=[], excluded_llm_metadata_keys=[], relationships={}, hash=..., text="Module name: base \n Docstring: None...") ]
"""
#### We can print the text of document
print(example1_docs[0].text[:500])
"""
Module name: base
Docstring: None
Class name: DocstringWalker
Docstring: A loader for docstring extraction and building structured documents from them.
Recursively walks a directory and extracts docstrings from each Python module - starting from the module
itself, then classes, then functions. Builds a graph of dependencies between the extracted docstrings.
Function name: load_data, In: DocstringWalker
Docstring: Load data from the specified code directory.
Additionally, after loading t
"""
#### Step 3: Feed documents into Llama Index
example1_index = VectorStoreIndex(
example1_docs, service_context=service_context
)
#### Step 4: Query the index
example1_qe = example1_index.as_query_engine(service_context=service_context)
#### Step 5: And start querying the index
print(
example1_qe.query(
"What are the main functions used by DocstringWalker? Describe each one in points."
).response
)
"""
1. load_data: This function loads data from a specified code directory and builds a dependency graph between the loaded documents. The graph is stored as an attribute of the class.
2. process_directory: This function processes a directory and extracts information from Python files. It returns a tuple containing a list of Document objects and a networkx Graph object. The Document objects represent the extracted information from Python files, and the Graph object represents the dependency graph between the extracted documents.
3. read_module_text: This function reads the text of a Python module given its path and returns the text of the module.
4. parse_module: This function parses a single Python module and returns a Document object with extracted information from the module.
5. process_class: This function processes a class node in the AST and adds relevant information to the graph. It returns a string representation of the processed class node and its sub-elements.
6. process_
FAQ
Common questions
Discussion
Questions & comments · 0
Sign In Sign in to leave a comment.