Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -161,7 +161,7 @@ build-docs: ## Build all documentation
@echo "Converting ipynb notebooks to md files..."
$(Q)MKDOCS_CI=true uv run python $(GIT_ROOT)/docs/ipynb_to_md.py
@echo "Building ragas documentation..."
$(Q)uv run --group docs mkdocs build
$(Q)MKDOCS_CI=false uv run --group docs mkdocs build

serve-docs: ## Build and serve documentation locally
$(Q)uv run --group docs mkdocs serve --dirtyreload
$(Q)MKDOCS_CI=false uv run --group docs mkdocs serve --dirtyreload
30 changes: 24 additions & 6 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -97,21 +97,39 @@ Available templates:

### Evaluate your LLM App

This is 5 main lines:
This is a simple example evaluating a summary for accuracy:

```python
from ragas import SingleTurnSample
from ragas.metrics import AspectCritic
import asyncio
from ragas.metrics.collections import AspectCritic
from ragas.llms import llm_factory

# Setup your LLM
llm = llm_factory("gpt-4o")

# Create a metric
metric = AspectCritic(
name="summary_accuracy",
definition="Verify if the summary is accurate and captures key information.",
llm=llm
)

# Evaluate
test_data = {
"user_input": "summarise given text\nThe company reported an 8% rise in Q3 2024, driven by strong performance in the Asian market. Sales in this region have significantly contributed to the overall growth. Analysts attribute this success to strategic marketing and product localization. The positive trend in the Asian market is expected to continue into the next quarter.",
"response": "The company experienced an 8% increase in Q3 2024, largely due to effective marketing strategies and product adaptation, with expectations of continued growth in the coming quarter.",
}
evaluator_llm = LangchainLLMWrapper(ChatOpenAI(model="gpt-4o"))
metric = AspectCritic(name="summary_accuracy",llm=evaluator_llm, definition="Verify if the summary is accurate.")
await metric.single_turn_ascore(SingleTurnSample(**test_data))

score = await metric.ascore(
user_input=test_data["user_input"],
response=test_data["response"]
)
print(f"Score: {score.value}")
print(f"Reason: {score.reason}")
```

> **Note**: Make sure your `OPENAI_API_KEY` environment variable is set.

Find the complete [Quickstart Guide](https://docs.ragas.io/en/latest/getstarted/evals)

## Want help in improving your AI application using evals?
Expand Down
Loading
Loading