Starstruk Logo MarkStarstruk

0.19.0

takahirom/arbigent0.19.0Feb 5, 2025by takahirom

AI Summary

Experimental features for system prompt optimization and automatic management of failed AI decision caches.

Key Highlights

  • Experimental JSONL storage for API request/response data for prompt optimization
  • Automatic removal of AI decision caches on test failure

New Features

  • Prompt optimization data collection
  • Failed cache removal

Full Release Notes

# Experiment to Optimize System Prompt  
We will store JSONL files in `arbigent-result/` containing requestBody and responseBody data. User feedback from the interface can also be recorded in `arbigent-result/` to enable AI-driven optimization.

<img width="864" alt="system-prompt-optimization-interface" src="https://github.com/user-attachments/assets/4e0e6037-497e-441d-b03a-3565e698d0c9" />

The current system prompt was developed through trial and error. We plan to implement the [COPRO optimization method](https://dspy.ai/deep-dive/optimizers/copro/) using collected request-response pairs as training data. While not yet implemented, we have [sample code](https://github.com/takahirom/arbigent/blob/takahirom/add-arbigent_prompt_optimization.ipynb/2025-02-05/notebooks/arbigent_prompt_optimization.ipynb) for prompt optimization.


# Failed Cache Removal  
We encountered recurring failures due to preserved AI decision caches after unsuccessful tests. This was resolved by automatically removing corresponding cache entries when tests fail.  

## What's Changed
* Add experimental methods for maintain ssot by @takahirom in https://github.com/takahirom/arbigent/pull/123
* Remove failed cache by @takahirom in https://github.com/takahirom/arbigent/pull/127
* Save API call jsonl and add feedback feature by @takahirom in https://github.com/takahirom/arbigent/pull/124


**Full Changelog**: https://github.com/takahirom/arbigent/compare/0.18.0...0.19.0