Oracle Tuxedo internals - blocking `tpacall()`
A service call in Oracle Tuxedo is performed by the
tpcall() function that actually is a wrapper around two other functions:
tpacall()- sending a request to the service
tpgetrply()- waiting for a response from the service
This post will focus on some implementation aspects of the
It is no secret that Oracle Tuxedo uses System V IPC messages queues for interprocess communication. But unlike POSIX message queue API, System V message queue API has no concept of timeouts: POSIX has
mq_timedsend() for sending a message while System V has only
msgsnd() function. So how does Oracle Tuxedo implement a timeout (
UBBCONFIG) for sending a request to the service?
A standard UNIX way of adding a timeout to blocking calls is by using
alarm() function that delivers
SIGALRM signal to the process after a number of seconds. Signal interrupts the execution of the system call and sets
EINTR. That was the way to write the code in the previous century when software was rather simple. Programming with signals was never easy and always error-prone. Add to that cases where software might already be using
alarm() for other purposes or it might have multiple threads (signal will be delivered to a “random” thread) and it becomes clear that
alarm() is not the way to go.
So how does Oracle Tuxedo implement the timeout for
tpacall()/msgsnd() calls? Well, it always calls
IPC_NOWAIT flag that causes the function to return with
errno equal to
EAGAIN when there is insufficient space available in the queue. If that happens, Oracle Tuxedo sleeps for one second(!!!) and repeats the call again and again until it either succeeds or blocking timeout occurs. And it does the first retry without sleeping - probably for cases when the queue is full for a tiny fraction of a second.
Clever, right? But this implementation has serious consequences for systems that handle several calls per second (not to mention thousands of calls per second):
- More than 1 request per second is sent to the queue.
- Due to fluctuations in processing time requests queue up.
- The queue becomes full, caller sleeps for one second.
- Caller no longer accepts requests itself so 1 second worth of requests queue up at the caller.
- Since no new requests are added to the first queue, all of the requests might be processed and the process goes idle and looks completely innocent.
A single hiccup that causes one queue to fill up will propagate up the caller stack and bring your Oracle Tuxedo application down on its knees.
I have tried to avoid that sleeping by calling
TPNOBLOCK and retrying failed calls until they succeed. But either spinning consumed too much CPU preventing other processes from getting CPU time or
tpacall() overhead of load balancing and serializing request produced worse results than just letting Oracle Tuxedo to sleep and retry.
The second option is to call
TPNOTIME flag - possibly blocking until the queue has enough space. It sounds bad on paper but the more I think about the actual algorithm under the non-blocking
msgsnd() the more I like this approach.
But the best solution, of course, is to use big queues and throttling to prevent queues from filling up and having to deal with this problem in the first place.