BEA Tuxedo™ - Unlock the Spawning Capability For Both Generic Applications and the Peoplesoft® Environment

Purpose

This white paper describes the methods and procedures to properly configure BEA Tuxedo to enable the server spawning and decaying feature in both generic Tuxedo applications and the Peoplesoft environment.

Why

While working with many current Tuxedo customers, I have found a disturbing trend: server spawning, which is a crucial feature of Tuxedo (a top 5 in my book) is not always implemented or even known or understood. I believe that it is a feature that should ALWAYS be configured for the application servers in any Tuxedo system because of its autonomous and fault-tolerant nature. Therefore, I am publishing this white paper not only to “glorify” the feature, but to disseminate the knowledge of how to implement it for their environment to as many customers as possible, because the requirement for Tuxedo systems to expand and grow in an autonomous fashion (i.e. without human monitoring and intervention) is mandatory in today’s market.

For the Peoplesoft customers (with BEA Tuxedo as the middleware) this feature is not configured correctly “out of the box”. Therefore, to properly enable server spawning, one must manually edit the configuration files directly (at the Tuxedo level), as it is impossible to properly and fully configure this feature through the existing interface of psadmin. This fact applies to all the default configurations (small, medium and large) in Peoplesoft versions 7.x and 8.x. By not having this properly configured, many Peoplesoft implementations experience inherent performance issues and their servers, although claiming to be “spawn” enabled, never really are.

White Paper Sections

Introduction
History
How Does it Really Work
Explanation of Parameters
How to Setup Server Spawning
Summary
References

Return to Top of Page

Introduction

Server spawning is one of the best features of BEA Tuxedo (and is sometimes its best kept secret) because it enables the product to dynamically grow (or shrink) the number of servers (i.e. processes) currently running based upon user-defined system loads. Thus, Tuxedo responds to peak system loads and/or “slow” days without the necessity of an operator or administrator. From mission critical to small applications, the result is that each system can benefit from the scalability and autonomous characteristics of this feature without human intervention. Tuxedo (rather than a person) constantly monitors the application requests and loads (i.e. the number of users sending in requests at any given moment) and makes decisions to spawn or decay the number of processes to meet the demand. By responding to these system loads, an application can act independently in a 24x7 capacity with an appropriate response to any fluctuations in the performance of the middle layer. The overall benefit to any system is improved performance, no disruption of service to the users during peak times and better-managed flow in the environment (all of which is job security for the genius Tuxedo Administrator who enables this feature).

For clarification, I would like to define some key terms, which will be used throughout this white paper to ensure comprehension. The word “server” refers to a compiled piece of C, Java or COBOL code written with the Tuxedo API. The key is that a server process is an executable that gets “launched” on a machine and typically has a PID (process ID) associated with it to identify it through the operating system level. Tuxedo runs on a multitude of operating system platforms including many flavors of UNIX (Sun, HP, IBM, SCO, RedHat, NCR, etc.) or Microsoft NT/2000. Server spawning is available on all the platforms. Server spawning concentrates on optimizing the “launching” behavior. This paper does not discuss tuning the operating system for these processes although there are many tips for accomplishing that goal – this paper focuses on scalability at the Tuxedo configuration level. Finally, the word “spawn” is associated with the activity that Tuxedo will start (fork/exec) another copy of a server executable on a particular machine when a threshold has been hit.

Return to Top of Page

History

Server spawning (also called server pool management), a feature which appeared in early releases of IMC Tuxedo release 4.2.2 and later appeared in BEA Tuxedo release 6.5, enables Tuxedo to start or stop servers in the applications based upon user-defined thresholds. For earlier releases of Tuxedo (prior to version 8.0), server spawning could only be configured for the RPC-based servers (both non and multi-threaded servers) and not for conversational servers. But, in Tuxedo version 8, conversational servers can now be configured with this option. The setup involves the use of an MSSQ (multiple servers, single queue) set, shown in Diagram 1 below. By comparison, a standard Tuxedo server configuration, known as a SSSQ (single server, single queue) model is shown in Diagram 2:

Diagram 1: MSSQ SET

Diagram 1

Diagram 2: SSSQ MODEL

In the early releases of the Tuxedo product, one single executable was “attached” to a single queue (known as the SSSQ model) where the server could dequeue its requests in FIFO and/or priority-based order. Years later, the MSSQ set was introduced to be able to allow more than 1 copy of a server the ability to “pull” off a “named” queue; hence an MSSQ set. The MSSQ set evolved the SSSQ model and was specifically designed to help improve the overall Tuxedo performance. It accomplishes this by decreasing the routing decisions involved in the load balancing algorithm because within one machine Tuxedo does not need to choose between multiple servers offering similar services with individual queues; instead it routes it to one queue and the next “non-busy” server handles the request. Hence, Tuxedo has greater efficiency within one machine.

This assumes that the load balancing algorithm is turned on (a MUST for any application as you get better Tuxedo performance, statistics, fault tolerance, etc.). It also increases the fault tolerant aspects of the application by “securing” the queue from lost messages or corruption. In an SSSQ model, if a single server process dies unexpectedly, there is a chance that the queue itself will be damaged, losing all messages currently queued. Whereas, if the server who dies is a member of an MSSQ set, the messages in the queue are safe based upon the locking mechanisms in place and processing continues. The only potential message loss is the one the server was processing when it died.

In Diagram 1, there are multiple copies of “Server A” pulling from the same “named” MSSQ set queue called “QUEUE1”. It is required that all servers which belong to an MSSQ set MUST offer the same set of services at all times. The MSSQ Set acts as a cohesive “unit” and allows the work to be distributed to the servers within it (whoever isn’t “busy” processes the next request off the queue). The “norm” is to configure multiple copies of the same executable working off the same queue but syntactically and operationally, the servers need not be the same actual executable just multiple executables offering the same services. It is important to note that generic, non-spawning MSSQ Sets can still bottleneck if the queue fills and the servers are all busy (the default behavior) despite the other features of the MSSQ set versus the SSSQ model.

The key to configuring and managing server spawning involves properly defining high and low water marks, which determine when Tuxedo spawns or decays the number of servers as well as the potential number of available servers. This involves collecting and analyzing data and statistics from an application. Data like the number of requests, frequency (volume), duration, system loads, data/message size, IPC queue size, etc. is of particular interest and can help to gauge and project the amount of servers and appropriate queue sizes required for the application (don’t forget to also allow for growth).

It is important to note that these threshold marks can be changed dynamically (at runtime) if the system changes its behavior through the Tuxedo Web-Based Monitor or via MIB calls. But, it is recommended to carefully set the values initially and monitor their success in a controlled, test environment and not to do the analysis in a vacuum – all configuration changes impact one another and other applications may affect the Tuxedo system performance and scalability. Hence, when gathering data and testing, it is the responsibility of the Tuxedo administrator to work with the appropriate team members (system, network, application, and database administrators as well as the application architecture or infrastructure groups) to properly define and establish the load capacity marks as the values may have outside influences.

Return to Top of Page

How Does It Really Work

The basic concept is that Tuxedo will monitor the number of requests (messages) or loads factors present (load factors are defined either at the service level in the configuration file or via the Tuxedo API) on each IPC message queue configured as an MSSQ set with server spawning and load balancing enabled. This information is stored in the Tuxedo Bulletin Board and at runtime, Tuxedo will compare the queue activities to the pre-determined high and low water marks. Once a threshold (i.e. water mark) is reached, Tuxedo reacts to the fluctuation by either spawning or decaying the number of servers in the system (again within some pre-set boundaries). This spawned server will then attach itself to the existing queue and help to begin to drain messages. And, in fact, Tuxedo may continue to launch servers (based upon the configuration) if there still is a backlog and the high water mark is still exceeded.

Return to Top of Page

Explanation of Parameters

First, one must ensure that the Tuxedo load balancing algorithm is enabled. Second, to setup the actual server spawning (in both a generic Tuxedo and Peoplesoft implementations), involves setting and tuning 5 parameters (at a minimum) for each server in the Tuxedo configuration file. In a generic Tuxedo environment, this file is generally called the UBBConfig file and it lives in the APPDIR directory. This file is compiled into a binary format (TUXCONFIG) using a command called tmloadcf for use (and safety) at runtime. In the Peopleosft environment, these parameters are contained in the Domain Template files, typically called PSAPPSRV.ubx which live in the $PS_HOME/appserv/domain (or %PS_HOME%\appserv\domain) directory.

For clarification, Peoplesoft uses a “generation” process to assemble their Tuxedo configuration file (see Diagram 3 below) using data in their .ubx, .val, and .cfg files plus values acquired through the psadmin facility. It is also recommended to make all changes for Tuxedo in the actual ubx files for each domain directly (by editing the file with your favorite editor) as many of the parameters are not accessible through psadmin. Then, once the parameters are properly configured, do not overwrite or edit these values through the psadmin configuration layer – simply, generate the binary file.

Return to Top of Page

Diagram 3: Peoplesoft Configuration File Generation Process

The UBBconfig file is divided into a series of sections from global parameters in the *RESOUCES section to the specific *ROUTING section. All server spawning parameters are contained in the *SERVERS section of the file. The five mandatory parameters for server spawning are as follows: RQADDR, REPLYQ, CLOPT (-p), MIN and MAX. The chart below gives a quick synopsis, syntax and definition of the parameters. There is more detailed information following the chart.

Chart: Required Server Spawning Parameters

Parameter	Definition/Syntax	Tips	Example(s)
RQADDR	“Names” a request queue to form an MSSQ set – it must be 30 characters or less and is typically contained in double quotes	If using different server names to form the queue set, both servers must live in the same UBBConfig GROUP (with the SRVGRP parameter)	RQADDR=”Queue1” RQADDR=”PSAPPSRVQ1”
REPLYQ	Enables a separate reply queue from the request queue (only to be used when servers nest, i.e. call each other and are a part of an MSSQ set)	For performance degradation reasons, it is BEST not to use reply queues unless the application demands it – for Peoplesoft users, please ensure that this option is turned off at all times**	REPLYQ=Y REPLYQ=N
CLOPT	Command line options for a server Format = “TUX Options -- Application Options” For server pool management, the TUX “-p” option is used -p [L] low_water, termination_time: high_water, creation_time (See below for a complete explanation of this parameter)	Easiest way to configure the high and low water marks is by the number of requests rather than the “load”. The termination time should be high (30 seconds to10 minutes) while the creation time should be very low (5 to 20 seconds). For Tuxedo version 8.x users, you can also specify the times to be 0 for instantaneous response.	CLOPT=”-A -p 2,30:5,10” CLOPT=”-A -p 4,60:10:20” CLOPT=”-A -p L 400,30:1000,10”
MIN	Minimum number of servers to boot. If the RQADDR parameter is set, the servers’ form an MSSQ set with the MIN parameter being the lowest bound on the number of servers working together in the set.	Please explicitly set this parameter as it defaults to 1. It is a MUST.	MIN=10 MIN=2
MAX	Maximum number of servers, which can be booted. If the RQADDR parameter and the “-p” option are set, then Tuxedo will automatically spawn or decay the number of servers booted at any time.	Please explicitly set this parameter as it defaults to MIN (which could be 1). It is a MUST.	MAX=100 MAX=20

Detailed Explanation of the “-p” arguments:

L:     If specified, Tuxedo monitors the queue load (i.e. load factors) rather than the number of messages on a queue (it only works in the SHM Model of Tuxedo with Load Balancing turned on). It is not recommended for use based upon these model limitations.

low water:     Establishes the low water mark for the queue set. It is either in terms of number of messages or load factors based upon the use of the “L” option above.

high water:    Establishes the high water mark for the queue set. It is either in terms of number of messages or load factors based upon the use of the “L” option above.

creation time:     The lag time (in seconds) Tuxedo waits before the spawning of servers

.
deletion time:     The lag time (in seconds) Tuxedo waits before the decaying of servers.

**Important Note: When the REPLYQ parameter is set to “N”, Tuxedo will produce a warning message when the UBBconfig file is loaded into the binary version with the tmloadcf command. The rationale for the warning message is safety and caution as it is more detrimental to an application which has servers nesting without reply queues because those servers can die and transactions lost rather than dealing with performance issues if reply queues are turned on inappropriately. Just be aware of this situation as you make changes to your configuration.

Return to Top of Page

How to Setup Server Spawning

The following is an excerpt from a UBBconfig file defining an MSSQ set with server spawning for a generic Tuxedo application:

GENERIC TUXEDO EXAMPLE:

MYSERVER SRVID=1 MIN=2 MAX=10 SRVGRP=MYGRP

CLOPT=”-A -p 2,60:5,10”

RQADDR=”QUEUE1” REPLYQ=N

The above example defines an MSSQ set with the server executable MYSERVER in MYGRP. The set has spawning enabled (via the use of the –p option) for QUEUE1 with the server boundaries set at 2 and 10 (MIN/MAX). The low water mark is at 2 messages and the high water mark is at 5 messages with the creation time at 10 seconds and the deletion time at 60 seconds.

When Tuxedo boots this MSSQ set example, 2 servers are started against the same queue (QUEUE1) as in Diagram 4 below:

Diagram 4:

While the number of messages on the queue remains 2 or under, the MSSQ set remains static after the initial boot. Likewise, until the high water mark of 5 messages is exceeded, no changes (spawning) will occur as in Diagram 5.

Diagram 5:

However, if the number of messages exceeds the high water mark of 5 requests for at least the creation time of 10 seconds, then Tuxedo spawns a new server. In fact, Tuxedo will continue to spawn servers in this fashion (assuming the high water condition does not decrease – i.e. the messages drops below 5) until the MAX threshold is encountered. Therefore, in the example, the application could have a maximum of 10 copies of MYSERVER working against the queue if the number of requests on the queue remains over 5 for an extended period of time. See Diagram 6 below:

Diagram 6:

Likewise, if the number of messages drops below the low water AND the MSSQ set is not back down to the MIN number of servers (considered to be the baseline), then the opposite occurs -- Tuxedo will decay the number of servers using the termination time of 60 seconds (in our example). So, if the queue only has 1 message (or less) on it for at least 60 seconds, then Tuxedo will deactivate 1 server and again will continue to do so until the MIN number of servers is reached.

PEOPLESOFT EXAMPLE:

For Peoplesoft users, it is important to note that the entries mentioned in the generic Tuxedo example will vary slightly from the .ubx files that you see (as Peoplesoft uses macros with either {} or <> to help define the parameters). In order to properly configure this environment, you will need to simply hard-code the parameters and delete the macros for the servers in your system in the ubx file for a particular Peoplesoft domain. An example is shown below:

EXISTING UBX ENTRY:

PSAPPSRV SRVGRP=APPSRV

SRVID=1

MIN={$PSAPPSRV\Min Instances}

MAX={$PSAPPSRV\Max Instances}

RQADDR="APPQ"

REPLYQ=Y

{QUERYSRV}

CLOPT="{$PSAPPSRV\Spawn Server} -A -- -C {CFGFILE} -D

{$Domain Settings\Domain ID} -S PSAPPSRV"

{QUERYSRV}

CHANGED UBX ENTRY:

PSAPPSRV SRVGRP=APPSRV

SRVID=1

MIN=2

MAX=20

RQADDR="APPQ"

REPLYQ=N

{QUERYSRV}

CLOPT="-A –p 2,120:5,10 -- -C {CFGFILE} -D

{$Domain Settings\Domain ID} -S PSAPPSRV"

{QUERYSRV}

From there, the Peoplesoft environment will behave exactly as described in the generic Tuxedo example. The only difference is that the Peoplesoft servers (PSAPPSRV, PSQUERYSRV, PSSAMSRV, etc.) are the ones being spawned.

Return to Top of Page

SUMMARY

As demonstrated by the facts in this white paper, the Tuxedo server spawning feature is key to launching any application. It should be obvious that the goal is to configure every server in an application as an MSSQ set with appropriate options so that the application NEVER encounters a situation where the queue set does not have room to grow or shrink, for that defeats the autonomous nature of the application and the ability of a system to respond to unknown loads without human help. Hence, it is crucial for all Tuxedo administrators to perform their analysis, configure Tuxedo to enable server spawning and monitor their application’s improved performance and scalability.

References

BEA Tuxedo Online Documentation (versions 6.5, 7.0 and 9.0 – http://edocs.bea.com/)
BEA Tuxedo 4.2.2 Manual Documentation
Peoplesoft Configuration and Administration Manual (version 7.x and 8.x)

PeopleSoft® is a registered trademark of the PeopleSoft Inc., not associated with Aurora Information Systems, Inc.
Tuxedo™ is a trademark of BEA Systems, Inc., not associated with Aurora Information Systems, Inc.

Return to Top of Page