p4p.client.thread.Context.put does not always work as expected when used inside a handler

[kurup]

I'm trying to replicate similar functionality to how a forward link works in a traditional IOC using p4p.client.thread.Context.put in the handler for a PV however, I've run into an issue with this not always working depending on the order in which PVs are created. A simple example of the problem is:

from p4p.server import Server
from p4p.server.thread import SharedPV
from p4p.client.thread import Context
from p4p.nt import NTScalar

class MyHandler:
    def put(self, pv, op):
        pv.post(op.value())
        op.done()

class MyHandlerWithPut:
    def put(self, pv, op):   
        pv.post(op.value())
        op.done()
        ctxt = Context('pva')
        ctxt.put("DEV:RW:DOUBLE1",{})

pv1 = SharedPV(nt=NTScalar('i'), initial=0.0, handler=MyHandler())
pv2 = SharedPV(nt=NTScalar('i'), initial=0.0, handler=MyHandler())
pv3 = SharedPV(nt=NTScalar('i'), initial=0.0, handler=MyHandler())
pv4 = SharedPV(nt=NTScalar('i'), initial=0.0, handler=MyHandler())
pv5 = SharedPV(nt=NTScalar('i'), initial=0.0, handler=MyHandlerWithPut())

Server.forever(providers=[{
    "DEV:RW:DOUBLE1": pv1, 
    "DEV:RW:DOUBLE2": pv2, 
    "DEV:RW:DOUBLE3": pv3, 
    "DEV:RW:DOUBLE4": pv4,
    "DEV:RW:DOUBLE5": pv5 
    }])

The idea is that if I do a put to DEV:RW:DOUBLE5 this then triggers a put to DEV:RW:DOUBLE1. However, if I do a put to DEV:RW:DOUBLE5 (from another terminal) the server gives this error:

Unexpected
Traceback (most recent call last):
  File "/home/myuser/.local/lib/python3.10/site-packages/p4p/client/thread.py", line 362, in put
    value, i = done.get(timeout=timeout)
  File "/usr/lib/python3.10/queue.py", line 179, in get
    raise Empty
_queue.Empty

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/home/myuser/.local/lib/python3.10/site-packages/p4p/server/thread.py", line 19, in _on_queue
    M(*args)
  File "/mnt/c/Users/mysuser/Software/p4p_for_isis/examples/basic_server_recipe-testing.py", line 40, in put
    ctxt.put(put_pv,{})
  File "/home/myuser/.local/lib/python3.10/site-packages/p4p/client/thread.py", line 365, in put
    raise TimeoutError()
TimeoutError
Error from WorkQueue w/ functools.partial(<function _on_queue at 0x7f771b1bca60>, <p4p._p4p.ServerOperation object at 0x7f771b1b3be0>, <bound method MyHandlerWithPut.put of <__main__.MyHandlerWithPut object at 0x7f771b1ec7f0>>, SharedPV(value=0), <p4p.server.raw.ServOpWrap object at 0x7f771b1eca30>)
Traceback (most recent call last):
  File "/home/myuser/.local/lib/python3.10/site-packages/p4p/util.py", line 56, in handle
    callable()
  File "/home/myuser/.local/lib/python3.10/site-packages/p4p/server/thread.py", line 27, in _on_queue
    op.done(error=str(err))
  File "src/p4p/_p4p.pyx", line 888, in p4p._p4p.ServerOperation.done
ValueError: Must provide error message

The odd thing is the above code works if I change the link to point to any of the other pvs (e.g. DEV:RW:DOUBLE2). This also works if I create another pv between pv1 and pv5. Diggging about in the code I found that I can fix this by changing the number of workers defined in the __init__ method of the class _DefaultWorkQueue in p4p.util, i.e. if I use def __init__(self, workers=5):

It looks like this problem occurs for the specific configuration of having 5 pvs in a row with the last one linking to the first. I'm not sure if this is a bug or if there is a different way to call a put to a pv from within a handler. This issue may also be related to #144

[mdavidsaver]

class MyHandlerWithPut:
def put(self, pv, op):
...
ctxt = Context('pva')
ctxt.put("DEV:RW:DOUBLE1",{})

Be careful. This will create an entirely new PVA client instance for each remote PUT. A substantial overhead considering the target PV is local, so there is no reason to go out over the network at all!

The onPut handler for one SharedPV is allowed to post() to other SharedPVs.

from p4p.server import Server
from p4p.server.thread import SharedPV
from p4p.nt import NTScalar

pv1 = SharedPV(nt=NTScalar('i'), initial=0.0)
pv2 = SharedPV(nt=NTScalar('i'), initial=0.0)

@pv2.put
def handle(thispv, op):
    thispv.post(op.value()) # aka. pv2.post(...)
    pv1.post(op.value()*2)
    op.done()

Server.forever(providers=[{
    "DEV:RW:DOUBLE1": pv1, 
    "DEV:RW:DOUBLE2": pv2, 
}])

[kurup]

Thanks for the suggestion, I didn't know you could call the post method for a different pv in the handler. This will fix the problem I'm having but now I need a way to pass into the handler the SharedPV object for PVs that exist on the same server. The PVs are created independently so I'd need to find out which PVs are on the same server after the server has started, since it's only then that all PVs will have been defined. That was part of the reason for using the Context.put method. Is there a way to get the SharedPV object of a PV that is local to a server once the server has been started? I couldn't find a good way to do this.

[mdavidsaver]

I didn't know you could call the post method for a different pv in the handler.

My goal for the APIs of P4P, and PVXS underneath, is fully re-entrant. Further P4P object should interact with the python cyclic garbage collector. So you should be able reasonably to tie these handler objects up into all kinds of loops.

I couldn't find a good way to do this.

I think this is more of a general python question.

One approach would be to use bound class members for the handler callbacks, then use self to reference the many SharedPV.