Validation Outcome (#431)

* start validation outcome changes

* fix gather_reasks for non-structured output

* lint fixes

* more lint fixes

* start test fixes, debug types

* fix tests

* fix types with overloads

* fix tests

* lint fixes

* lint fixes

* fix tests

* lint fixes

* switch to generics for ValidationOutcome

* allow destructuring

* remove None from generic type

* init commit, changes to handle error in guard

* handle error a layer deeper

* update return in text2sql

* remove extra fx in validation outcome

* use error instead of exception

* remove print statements plus lint

* fix type

* fix typing while maintaining type hinting

* fix other type issues

* autoformat

* lint fixes

* test fixes

* autoformat

* type fixes

* lint fix

* unused import

* uncomment test parameters

* merge/type fixes

* guard: Allow calling parse with preconfigured num_reasks (#423)

* Cron nb (#425)

* install deps + pkg for nb runs

* lock nb runner to 3.11.x

* use cohere api key from environ

* ref env vars for cohere + openai

* fix bad merge in code originally from validators.py (#427)

* fix bad merge in code originally from validators.py

* lint fixes

* bump version (#428)

* update notebooks

* Setup passed password (#429)

* use pypi pass from env

* upgrade pip before installing deps

* pass pypi pass explicitly

* use environ competently

* list -> List

* lint and test fixes

* autoformat

* lint and type fix

* fix test

* fix llm_output type

* ' -> "

* fix tests

* lint fixes

* fix notebooks again

* fix docs

* debug

* validated_response -> validated_output

---------

Co-authored-by: Nefertiti  Rogers <[email protected]>
Co-authored-by: Nefertiti  Rogers <[email protected]>
Co-authored-by: rafael <[email protected]>
Co-authored-by: zsimjee <[email protected]>

impl and refactor

autoformat and lint fixes

lint fixes

fix reask merging, harmonize Call and ValidationOutcome

mark test as TODO

force clean merge

Validation Outcome (#431)

* start validation outcome changes

* fix gather_reasks for non-structured output

* lint fixes

* more lint fixes

* start test fixes, debug types

* fix tests

* fix types with overloads

* fix tests

* lint fixes

* lint fixes

* fix tests

* lint fixes

* switch to generics for ValidationOutcome

* allow destructuring

* remove None from generic type

* init commit, changes to handle error in guard

* handle error a layer deeper

* update return in text2sql

* remove extra fx in validation outcome

* use error instead of exception

* remove print statements plus lint

* fix type

* fix typing while maintaining type hinting

* fix other type issues

* autoformat

* lint fixes

* test fixes

* autoformat

* type fixes

* lint fix

* unused import

* uncomment test parameters

* merge/type fixes

* guard: Allow calling parse with preconfigured num_reasks (#423)

* Cron nb (#425)

* install deps + pkg for nb runs

* lock nb runner to 3.11.x

* use cohere api key from environ

* ref env vars for cohere + openai

* fix bad merge in code originally from validators.py (#427)

* fix bad merge in code originally from validators.py

* lint fixes

* bump version (#428)

* update notebooks

* Setup passed password (#429)

* use pypi pass from env

* upgrade pip before installing deps

* pass pypi pass explicitly

* use environ competently

* list -> List

* lint and test fixes

* autoformat

* lint and type fix

* fix test

* fix llm_output type

* ' -> "

* fix tests

* lint fixes

* fix notebooks again

* fix docs

* debug

* validated_response -> validated_output

---------

Co-authored-by: Nefertiti  Rogers <[email protected]>
Co-authored-by: Nefertiti  Rogers <[email protected]>
Co-authored-by: rafael <[email protected]>
Co-authored-by: zsimjee <[email protected]>

many test fixes, many more to come

guardrails/run.py

@@ -692,7 +692,6 @@ async def async_run(
             missing_keys = verify_metadata_requirements(
                 self.metadata, self.output_schema.root_datatype
             )
-
             if missing_keys:
                 raise ValueError(
                     f"Missing required metadata keys: {', '.join(missing_keys)}"

tests/integration_tests/test_data_validation.py

@@ -68,7 +68,6 @@ def test_choice_validation(llm_output, raises, fails):
         assert result.validated_output is not None
         assert not isinstance(result.validated_output, ReAsk)
 
-
 @pytest.mark.parametrize("llm_output, raises, fails", test_cases)
 def test_choice_validation_pydantic(llm_output, raises, fails):
     class Fight(BaseModel):