WIP: Feature/Add Langfuse Integration for User Feedback Tracking

[bnodir]

Purpose

This PR implements Langfuse integration for systematic tracking and analysis of user feedback on chat responses, enabling better monitoring and insights into user interactions.

Feedback.feature.with.Langfuse_2025-01-12.111127.Clipchamp.mp4

🔄 Key Changes

• Added Langfuse SDK integration for feedback tracking
• Implemented unified feedback handling system
• Enhanced feedback UI with success/error notifications
• Added custom feedback state management hook
• Improved error handling and logging mechanisms

🛠️ Setup Required

Add to your .env file:

# Langfuse Configuration
LANGFUSE_HOST="https://cloud.langfuse.com"
LANGFUSE_PUBLIC_KEY="<your-langfuse-public-key>"
LANGFUSE_SECRET_KEY="<your-langfuse-secret-key>"

✅ Testing Checklist

1. Langfuse credentials configuration
2. Positive/negative feedback submission
3. Langfuse data verification
   • Verify correct feedback score values (1 for👍/0 for👎)
   • Validate metadata (trace_id, timestamp, user_oid)
   • Check token count accuracy
   • Verify prompt and completion data
   • Confirm event types are properly labeled
   • Test response metrics(latency and etc)
4. Network error handling
5. Local history functionality
6. Multi-language support for new strings
7. CosmosDB Integration Testing
   • Verify feedback sync with CosmosDB enabled
   • Test history retrieval with CosmosDB
   • Validate trace_id correlation in CosmosDB
   • Check feedback persistence across sessions
8. Verify optional feature functionality
   • Test with Langfuse disabled
   • Test with invalid credentials
   • Test fallback to local storage

📝 Important Notes

• Requires Langfuse account setup for analytics features
• Dual storage implementation:
  • Local: Always stored in IndexedDB/browser storage
  • Langfuse: Optional remote analytics storage when configured
• Uses trace IDs for feedback correlation
• Needs additional testing for non-Langfuse mode
• Optional feature - system continues to work with local storage only

Please verify your Langfuse credentials before deploying to production.

Does this introduce a breaking change?

When developers merge from main and run the server, azd up, or azd deploy, will this produce an error?
If you're not sure, try it out on an old environment.

[ ] Yes
[ ] No

Verify if this produces errors when:

• Running local dev server
• Running azd up
• Running azd deploy
• Running in existing environments

Does this require changes to learn.microsoft.com docs?

This repository is referenced by this tutorial
which includes deployment, settings and usage instructions. If text or screenshot need to change in the tutorial,
check the box below and notify the tutorial author. A Microsoft employee can do this for you if you're an external contributor.

[ ] Yes
[x] No (?)

Type of change

[ ] Bugfix
[x] Feature
[ ] Code style update (formatting, local variables)
[ ] Refactoring (no functional changes, no api changes)
[ ] Documentation content changes
[ ] Other... Please describe:

Code quality checklist

See CONTRIBUTING.md for more details.

• The current tests all pass (python -m pytest).
• I added tests that prove my fix is effective or that my feature works
• I ran python -m pytest --cov to verify 100% coverage of added lines
• I ran python -m mypy to check for type errors
• I either used the pre-commit hooks or ran ruff and black manually on my code.

[bnodir]

Will discard changes

[pamelafox]

Thanks for the PR!

We've discussed potential Langfuse integration here:
#1300
The issue is that we'd need to integrate it in a way that was completely optional, since many developers would not use langfuse.
Have you considered approaches that would make Langfuse optional? It doesn't appear optional here, given the way Langfuse is constructed in the global space.
Ideally it'd be like Azure monitor, constructed only based on an environment variable inside the client setup code.

Also, I'm very excited to hear that you've added a feedback store, has that been added in such a way that it can be used with multiple backing options, such as Cosmos DB? I haven't dug into that part of your code yet.

[bnodir]

Suggested change

	from openai import AsyncAzureOpenAI, AsyncOpenAI
	from langfuse.openai import AsyncAzureOpenAI, AsyncOpenAI

Token usage and other metrics tracking implementation
References:

• Issue: #1300 - Tracking Problem
• Solution: #1300 - Resolution

While token usage tracking and other related functionalities are successfully implemented, feedback tracking does not work beyond this stage.