Load and Process Data with Lilac

The Lilac Reader lets you bring cleaned, enriched data back from Lilac -- an open-source tool for analyzing, enriching, and cleaning unstructured data with AI -- into LlamaIndex documents. The workflow runs both directions: load data into LlamaIndex with any loader (for example ArxivReader), push it into a Lilac dataset for cleaning, labeling, or structuring, then pull the cleaned dataset back into LlamaIndex documents with LilacReader.

Use it when you have already cleaned, labeled, or structured a dataset in Lilac and need it back as LlamaIndex documents for indexing or querying. It assumes you already have Lilac running locally with a project directory and dataset set up, so it is not usable without first running Lilac and creating that project.

Install with:

pip install llama-index-readers-papers

pip install llama-index-readers-lilac

Getting data into Lilac (from any LlamaIndex loader, here ArxivReader):

import lilac as ll

from llama_index.readers.papers import ArxivReader

loader = ArxivReader()
documents = loader.load_data(search_query="au:Karpathy")

ll.set_project_dir("./data")

ll.create_dataset(
    config=ll.DatasetConfig(
        namespace="local",
        name="arxiv-karpathy",
        source=ll.LlamaIndexDocsSource(
            documents=documents
        ),
    )
)

ll.start_server(project_dir="./data")

Bringing cleaned data back into LlamaIndex:

from llama_index.core import VectorStoreIndex, download_loader

from llama_index.readers.lilac import LilacReader

loader = LilacReader()
documents = loader.load_data(
    project_dir="~/my_project",
    dataset="local/arxiv-karpathy",
)

index = VectorStoreIndex.from_documents(documents)

index.query("How are ImageNet labels validated?")

LilacReader.load_data takes a project_dir (your local Lilac project directory) and a dataset name (the namespace and name of the dataset within that project).

Developers who use Lilac to clean, structure, or label their data and need the cleaned result back inside a LlamaIndex or LangChain pipeline.