How can i configure Caching on @pytest.mark.langsmith

LangSmith Product Help Observability & Evals
1

HI Team
I recently switch to using the

@pytest.mark.langsmith

annotation in my pytest
Which is now caching llm calls, which is great as the llm costs on unit tests was becoming quite painful

However it also seems to cache other api calls (non llm)
like

  • aws api gateway rest api calls
  • posts to cognito idp for token exchange

is there a way i can disable the caching on these?

thanks
Dharshans

2

Hey @darthShana,

I’m working on a way to allow configuration here. Will update when ready!

Best,

Jacob

3

thanks @jacoblee93

If i can also add that im getting a lot of casset errors to do with

‘set being modified after creation‘

this maybe related..

thanks

Dharshana

4

This is now live in 0.4.10:

Let me know how it goes for you!

5

hi @jacoblee9315

im still getting these

request = <FixtureRequest for <Function test_generate_tax_statement>>
test_args = ()
test_kwargs = {'mock_user': UserInfo(user_id='test', email='test', first_name='Dharshana', last_name='Ratnayake', company_name=None,...'F_Rental_Statement.pdf'}, 'transaction_type': 'property management'}, reference_docs=[], linked_transaction={}), ...]}
i = 0
repetition_extra = {'cache': '/tmp/test-cache/cassettes', 'cached_hosts': ['api.openai.com', 'https://api.anthropic.com'], 'client': None, 'id': None, ...}

    @functools.wraps(func)
    def wrapper(*test_args: Any, request: Any = None, **test_kwargs: Any):
        if disable_tracking:
            return func(*test_args, **test_kwargs)
    
        # Run test multiple times for repetitions
        for i in range(repetitions):
            repetition_extra = langtest_extra.copy()
>           _run_test(
                func,
                *test_args,
                pytest_request=request,
                **test_kwargs,
                langtest_extra=repetition_extra,
            )

/root/.cache/pypoetry/virtualenvs/my-agent-HzH--rk3-py3.12/lib/python3.12/site-packages/langsmith/testing/_internal.py:388: 
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 
/root/.cache/pypoetry/virtualenvs/my-agent-HzH--rk3-py3.12/lib/python3.12/site-packages/langsmith/testing/_internal.py:995: in _run_test
    ls_utils.with_optional_cache(
/root/.pyenv/versions/3.12.11/lib/python3.12/contextlib.py:144: in __exit__
    next(self.gen)
/root/.cache/pypoetry/virtualenvs/my-agent-HzH--rk3-py3.12/lib/python3.12/site-packages/langsmith/utils.py:579: in with_optional_cache
    with with_cache(path, ignore_hosts, allow_hosts):
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
/root/.pyenv/versions/3.12.11/lib/python3.12/contextlib.py:144: in __exit__
    next(self.gen)
/root/.cache/pypoetry/virtualenvs/my-agent-HzH--rk3-py3.12/lib/python3.12/site-packages/langsmith/utils.py:567: in with_cache
    with ls_vcr.use_cassette(cache_file):
         ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
/root/.cache/pypoetry/virtualenvs/my-agent-HzH--rk3-py3.12/lib/python3.12/site-packages/vcr/cassette.py:98: in __exit__
    next(self.__finish, None)
/root/.cache/pypoetry/virtualenvs/my-agent-HzH--rk3-py3.12/lib/python3.12/site-packages/vcr/cassette.py:57: in _patch_generator
    with contextlib.ExitStack() as exit_stack:
         ^^^^^^^^^^^^^^^^^^^^^^
/root/.pyenv/versions/3.12.11/lib/python3.12/contextlib.py:610: in __exit__
    raise exc_details[1]
/root/.pyenv/versions/3.12.11/lib/python3.12/contextlib.py:595: in __exit__
    if cb(*exc_details):
       ^^^^^^^^^^^^^^^^
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ 

self = <vcr.patch.ConnectionRemover object at 0x7f5ce596dd00>
args = (None, None, None)
pool = <urllib3.connectionpool.HTTPSConnectionPool object at 0x7f5ce9a332f0>
connections = {<vcr.patch.VCRRequestsHTTPSConnection/tmp/test-cache/cassettes/29ea9c13-b100-4232-836b-68d3492c59ef.yaml object at 0x...CRRequestsHTTPSConnection/tmp/test-cache/cassettes/29ea9c13-b100-4232-836b-68d3492c59ef.yaml object at 0x7f5ce592a8d0>}
readd_connections = []

    def __exit__(self, *args):
        for pool, connections in self._connection_pool_to_connections.items():
            readd_connections = []
            while pool.pool and not pool.pool.empty() and connections:
                connection = pool.pool.get()
                if isinstance(connection, self._connection_class):
                    connections.remove(connection)
                    connection.close()
                else:
                    readd_connections.append(connection)
            for connection in readd_connections:
                pool._put_conn(connection)
>           for connection in connections:
                              ^^^^^^^^^^^
E           RuntimeError: Set changed size during iteration

/root/.cache/pypoetry/virtualenvs/my-agent-HzH--rk3-py3.12/lib/python3.12/site-packages/vcr/patch.py:391: RuntimeError
6

i think its to do with running multiple tests..

7

Hmm. If you have a way I can try to repro can you share it? Sounds like running things in parallel?

8

i can add you as a collaborator on GitHub, if you have a username..

im running

      - poetry run pytest --junitxml=test-reports/pytest-report.xml

i can try running one at a time

9

okey actually think it may be resolved. It looks like this happens when test fail for other reasons..

I will keep monitoring, and raise a different issue if if it comes up again.

thanks @jacoblee9315 for your amazing work