Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[WIP] Notebook testing changes #1606

Open
wants to merge 17 commits into
base: main
Choose a base branch
from

Merge branch 'main' into corey/nb-tests

231376e
Select commit
Loading
Failed to load commit list.
Open

[WIP] Notebook testing changes #1606

Merge branch 'main' into corey/nb-tests
231376e
Select commit
Loading
Failed to load commit list.
Azure Pipelines / PR Validation Eval failed Dec 4, 2024 in 16m 49s

Build #20241204.16 had test failures

Details

Tests

  • Failed: 15 (3.90%)
  • Passed: 370 (96.10%)
  • Other: 0 (0.00%)
  • Total: 385

Annotations

Check failure on line 317 in Build log

@azure-pipelines azure-pipelines / PR Validation Eval

Build log #L317

2 test(s) failed, 77 test(s) collected.

Check failure on line 610 in Build log

@azure-pipelines azure-pipelines / PR Validation Eval

Build log #L610

Bash exited with code '2'.

Check failure on line 326 in Build log

@azure-pipelines azure-pipelines / PR Validation Eval

Build log #L326

2 test(s) failed, 77 test(s) collected.

Check failure on line 791 in Build log

@azure-pipelines azure-pipelines / PR Validation Eval

Build log #L791

Bash exited with code '2'.

Check failure on line 1 in tests/unit/test_feedback.py::TestFeedbackEval::test_skipeval_all

@azure-pipelines azure-pipelines / PR Validation Eval

tests/unit/test_feedback.py::TestFeedbackEval::test_skipeval_all

[gw0] linux -- Python 3.10.15 /opt/hostedtoolcache/Python/3.10.15/x64/bin/python3.10

self = <tests.unit.test_feedback.TestFeedbackEval testMethod=test_skipeval_all>

    def test_skipeval_all(self) -> None:
        """Test the SkipEval capability for when all evals are skipped"""
    
        f = core_feedback.Feedback(imp=test_feedbacks.skip_if_odd).on(
            val=select_schema.Select.RecordCalls.somemethod.args.num[:]
        )
    
        # Create source data that looks like real source data for a record
        # collected from a real app. Store some integers in a place that
        # corresponds to app call to `somemethod`, keyword argument `num`.
        source_data = {
            "__record__": {"app": {"somemethod": {"args": {"num": [1, 3, 5]}}}}
        }
    
        res = f.run(source_data=source_data)
    
>       self.assertIsInstance(res.result, float)
E       AssertionError: None is not an instance of <class 'float'>

tests/unit/test_feedback.py:65: AssertionError
Raw output
/opt/hostedtoolcache/Python/3.10.15/x64/lib/python3.10/unittest/case.py:675: AssertionError: None is not an instance of <class 'float'>

Check failure on line 1 in tests/unit/test_feedback.py::TestFeedbackConstructors::test_nonglobal_feedback_functions

@azure-pipelines azure-pipelines / PR Validation Eval

tests/unit/test_feedback.py::TestFeedbackConstructors::test_nonglobal_feedback_functions

[gw0] linux -- Python 3.10.15 /opt/hostedtoolcache/Python/3.10.15/x64/bin/python3.10

self = <sqlalchemy.engine.base.Connection object at 0x7f88cf151a20>
dialect = <sqlalchemy.dialects.sqlite.pysqlite.SQLiteDialect_pysqlite object at 0x7f88cfff1570>
context = <sqlalchemy.dialects.sqlite.base.SQLiteExecutionContext object at 0x7f88cf151f30>
statement = <sqlalchemy.dialects.sqlite.base.SQLiteCompiler object at 0x7f88cfff39a0>
parameters = [('record_hash_b29974819c4ffe3aea6d1a6d43a9819b',)]

    def _exec_single_context(
        self,
        dialect: Dialect,
        context: ExecutionContext,
        statement: Union[str, Compiled],
        parameters: Optional[_AnyMultiExecuteParams],
    ) -> CursorResult[Any]:
        """continue the _execute_context() method for a single DBAPI
        cursor.execute() or cursor.executemany() call.
    
        """
        if dialect.bind_typing is BindTyping.SETINPUTSIZES:
            generic_setinputsizes = context._prepare_set_input_sizes()
    
            if generic_setinputsizes:
                try:
                    dialect.do_set_input_sizes(
                        context.cursor, generic_setinputsizes, context
                    )
                except BaseException as e:
                    self._handle_dbapi_exception(
                        e, str(statement), parameters, None, context
                    )
    
        cursor, str_statement, parameters = (
            context.cursor,
            context.statement,
            context.parameters,
        )
    
        effective_parameters: Optional[_AnyExecuteParams]
    
        if not context.executemany:
            effective_parameters = parameters[0]
        else:
            effective_parameters = parameters
    
        if self._has_events or self.engine._has_events:
            for fn in self.dispatch.before_cursor_execute:
                str_statement, effective_parameters = fn(
                    self,
                    cursor,
                    str_statement,
                    effective_parameters,
                    context,
                    context.executemany,
                )
    
        if self._echo:
            self._log_info(str_statement)
    
            stats = context._get_cache_stats()
    
            if not self.engine.hide_parameters:
                self._log_info(
                    "[%s] %r",
                    stats,
                    sql_util._repr_params(
                        effective_parameters,
                        batches=10,
                        ismulti=context.executemany,
                    ),
                )
            else:
                self._log_info(
                    "[%s] [SQL parameters hidden due to hide_parameters=True]",
                    stats,
                )
    
        evt_handled: bool = False
        try:
            if context.execute_style is ExecuteStyle.EXECUTEMANY:
                effective_parameters = cast(
                    "_CoreMultiExecuteParams", effective_parameters
                )
                if self.dialect._has_events:
                    for fn in self.dialect.dispatch.do_executemany:
                        if fn(
                            cursor,
                            str_statement,
                            effective_parameters,
                            context,
                        ):
                            evt_handled = True
                            break
                if not evt_handled:
                    self.dialect.do_executemany(
                        cursor,
                        str_statement,
                        effective_parameters,
                        context,
                    )
            elif not effective_parameters and context.no_parameters:
                if self.dialect._has_events:
                    for fn in self.dialect.dispatch.do_execute_no_params:
                        if fn(cursor, str_statement, cont
Raw output
/opt/hostedtoolcache/Python/3.10.15/x64/lib/python3.10/site-packages/sqlalchemy/engine/default.py:941: sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) no such table: trulens_records

Check failure on line 1 in tests/unit/test_feedback.py::TestFeedbackConstructors::test_global_feedback_functions

@azure-pipelines azure-pipelines / PR Validation Eval

tests/unit/test_feedback.py::TestFeedbackConstructors::test_global_feedback_functions

[gw1] linux -- Python 3.9.20 /opt/hostedtoolcache/Python/3.9.20/x64/bin/python3.9

self = <sqlalchemy.engine.base.Connection object at 0x7fd8d9e953a0>
dialect = <sqlalchemy.dialects.sqlite.pysqlite.SQLiteDialect_pysqlite object at 0x7fd8d9e8c3a0>
context = <sqlalchemy.dialects.sqlite.base.SQLiteExecutionContext object at 0x7fd8d9fc6f40>
statement = <sqlalchemy.dialects.sqlite.base.SQLiteCompiler object at 0x7fd8d9fc6e20>
parameters = [('app_hash_baf7b2cb6402e84fa3b0b3a028d4bf65', 1, 0)]

    def _exec_single_context(
        self,
        dialect: Dialect,
        context: ExecutionContext,
        statement: Union[str, Compiled],
        parameters: Optional[_AnyMultiExecuteParams],
    ) -> CursorResult[Any]:
        """continue the _execute_context() method for a single DBAPI
        cursor.execute() or cursor.executemany() call.
    
        """
        if dialect.bind_typing is BindTyping.SETINPUTSIZES:
            generic_setinputsizes = context._prepare_set_input_sizes()
    
            if generic_setinputsizes:
                try:
                    dialect.do_set_input_sizes(
                        context.cursor, generic_setinputsizes, context
                    )
                except BaseException as e:
                    self._handle_dbapi_exception(
                        e, str(statement), parameters, None, context
                    )
    
        cursor, str_statement, parameters = (
            context.cursor,
            context.statement,
            context.parameters,
        )
    
        effective_parameters: Optional[_AnyExecuteParams]
    
        if not context.executemany:
            effective_parameters = parameters[0]
        else:
            effective_parameters = parameters
    
        if self._has_events or self.engine._has_events:
            for fn in self.dispatch.before_cursor_execute:
                str_statement, effective_parameters = fn(
                    self,
                    cursor,
                    str_statement,
                    effective_parameters,
                    context,
                    context.executemany,
                )
    
        if self._echo:
            self._log_info(str_statement)
    
            stats = context._get_cache_stats()
    
            if not self.engine.hide_parameters:
                self._log_info(
                    "[%s] %r",
                    stats,
                    sql_util._repr_params(
                        effective_parameters,
                        batches=10,
                        ismulti=context.executemany,
                    ),
                )
            else:
                self._log_info(
                    "[%s] [SQL parameters hidden due to hide_parameters=True]",
                    stats,
                )
    
        evt_handled: bool = False
        try:
            if context.execute_style is ExecuteStyle.EXECUTEMANY:
                effective_parameters = cast(
                    "_CoreMultiExecuteParams", effective_parameters
                )
                if self.dialect._has_events:
                    for fn in self.dialect.dispatch.do_executemany:
                        if fn(
                            cursor,
                            str_statement,
                            effective_parameters,
                            context,
                        ):
                            evt_handled = True
                            break
                if not evt_handled:
                    self.dialect.do_executemany(
                        cursor,
                        str_statement,
                        effective_parameters,
                        context,
                    )
            elif not effective_parameters and context.no_parameters:
                if self.dialect._has_events:
                    for fn in self.dialect.dispatch.do_execute_no_params:
                        if fn(cursor, str_statement, conte
Raw output
/opt/hostedtoolcache/Python/3.9.20/x64/lib/python3.9/site-packages/sqlalchemy/engine/default.py:941: sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) no such table: trulens_apps

Check failure on line 1 in tests/unit/test_feedback.py::TestFeedbackConstructors::test_global_unsupported

@azure-pipelines azure-pipelines / PR Validation Eval

tests/unit/test_feedback.py::TestFeedbackConstructors::test_global_unsupported

[gw1] linux -- Python 3.9.20 /opt/hostedtoolcache/Python/3.9.20/x64/bin/python3.9

self = <sqlalchemy.engine.base.Connection object at 0x7fd8d3d57790>
dialect = <sqlalchemy.dialects.sqlite.pysqlite.SQLiteDialect_pysqlite object at 0x7fd8d9e8c3a0>
context = <sqlalchemy.dialects.sqlite.base.SQLiteExecutionContext object at 0x7fd8d3d57df0>
statement = <sqlalchemy.dialects.sqlite.base.SQLiteCompiler object at 0x7fd8d9fc6e20>
parameters = [('app_hash_baf7b2cb6402e84fa3b0b3a028d4bf65', 1, 0)]

    def _exec_single_context(
        self,
        dialect: Dialect,
        context: ExecutionContext,
        statement: Union[str, Compiled],
        parameters: Optional[_AnyMultiExecuteParams],
    ) -> CursorResult[Any]:
        """continue the _execute_context() method for a single DBAPI
        cursor.execute() or cursor.executemany() call.
    
        """
        if dialect.bind_typing is BindTyping.SETINPUTSIZES:
            generic_setinputsizes = context._prepare_set_input_sizes()
    
            if generic_setinputsizes:
                try:
                    dialect.do_set_input_sizes(
                        context.cursor, generic_setinputsizes, context
                    )
                except BaseException as e:
                    self._handle_dbapi_exception(
                        e, str(statement), parameters, None, context
                    )
    
        cursor, str_statement, parameters = (
            context.cursor,
            context.statement,
            context.parameters,
        )
    
        effective_parameters: Optional[_AnyExecuteParams]
    
        if not context.executemany:
            effective_parameters = parameters[0]
        else:
            effective_parameters = parameters
    
        if self._has_events or self.engine._has_events:
            for fn in self.dispatch.before_cursor_execute:
                str_statement, effective_parameters = fn(
                    self,
                    cursor,
                    str_statement,
                    effective_parameters,
                    context,
                    context.executemany,
                )
    
        if self._echo:
            self._log_info(str_statement)
    
            stats = context._get_cache_stats()
    
            if not self.engine.hide_parameters:
                self._log_info(
                    "[%s] %r",
                    stats,
                    sql_util._repr_params(
                        effective_parameters,
                        batches=10,
                        ismulti=context.executemany,
                    ),
                )
            else:
                self._log_info(
                    "[%s] [SQL parameters hidden due to hide_parameters=True]",
                    stats,
                )
    
        evt_handled: bool = False
        try:
            if context.execute_style is ExecuteStyle.EXECUTEMANY:
                effective_parameters = cast(
                    "_CoreMultiExecuteParams", effective_parameters
                )
                if self.dialect._has_events:
                    for fn in self.dialect.dispatch.do_executemany:
                        if fn(
                            cursor,
                            str_statement,
                            effective_parameters,
                            context,
                        ):
                            evt_handled = True
                            break
                if not evt_handled:
                    self.dialect.do_executemany(
                        cursor,
                        str_statement,
                        effective_parameters,
                        context,
                    )
            elif not effective_parameters and context.no_parameters:
                if self.dialect._has_events:
                    for fn in self.dialect.dispatch.do_execute_no_params:
                        if fn(cursor, str_statement, conte
Raw output
/opt/hostedtoolcache/Python/3.9.20/x64/lib/python3.9/site-packages/sqlalchemy/engine/default.py:941: sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) no such table: trulens_apps