EAServer Performance Tuning Techniques

The HTTP thread settings allow you to balance memory resources. A maximum value set too high needlessly uses memory resources. To tune this setting, monitor the total number of hits listed in the httpstat.dat file for indications of a heavily loaded server, and adjust the maximum thread setting as necessary.

Setting this number too high will result in thrashing, with many threads waiting for a slice of CPU time. As a general guideline, consider a limit of 100 clients per CPU on Solaris, and 75 clients per CPU on Windows.

· com.sybase.jaguar.server.http.maxthreads
· com.sybase.jaguar.server.http.numthreads

Section 10 System-Level Tuning and Sizing

"HTTP Config" in EAServer System Administration Guide

Extra threads are needed for service components, thread manager and message service. If you are not calculating precisely, a guideline is to add another 50 threads.

Important Note: See component Bind Thread property -- components with this setting require twice as many threads.

Stack size determines how many client requests can be served simultaneously. If there are a large number of clients utilizing the server, you may want to reduce the default stack size, which is the amount of memory reserved for the call stack associated with each thread.

For Unix, set the stack size using the com.sybase.jaguar.server.stacksize property.
For NT you need to modify the jagsrv.exe using editbin.

You can specify the minimum and maximum JVM heap size with com.sybase.jaguar.server.jvm.maxHeapSize and com.sybase.jaguar.server.jvm.minHeapSize

(This property is valid for any version of the JVM)

"Repository Properties Reference" in EAServer System Administration Guide

Memory Management within Java Processes

NT & Solaris flags

• -Xms - minimum heap size
  Always increase the -Xms to a reasonable size (such as at least 64m) so the JVM does not have to spend time allocating memory.
• -Xmx - maximum heap size
  If you have the memory to spare, set -Xms to the same size as -Xmx.
  (Note: On Solaris, the maximum value for -Xmx is 4g. In order to use the maximum value on Solaris, check shmmax in /etc/system that
  shmsys:shminfo_shmmax=0xffffffff
• Garbage Collection (GC)
  These flags are for tuning the Garbage Collections memory allocations and are highly dependent on the behavior of your application. They should only be set after collecting statistics on the GC behavior of your application.
  • -XX:NewSize - minimum size of the young generation
    
  • -XX:MaxNewSize - maximum size of the young generation
    
  • -XX:NewRatio - ratio between young and old generation
    
  • -XX:SurvivorRatio - ratio of survivor space with respect to young generation, there are 2 survivor spaces in the young generation
  • Setting NewSize and MaxNewSize to each other bounds the size of the young generation like -Xms and -Xmx for the heap size. NewRatio is a more coarse level option to do the same thing. The SurvivorRatio is set to 25 by default and should be sufficient. You should set the NewSize and MaxNewSize to half the size of -Xmx if you observe a lot of regular GCs happening and want to delay a full GC as much as possible. Your application will run more smoothly for the majority of the time and you only pay for the one full GC instead of many GCS throughout your application.
  • -XX:ThreadStackSize
    If you know that EAServer will be creating a lot of threads, like when servicing a lot of clients or doing operations like creating a lot of objects or beans, then it might be of benefit to lower the stack size so that they don't take up a lot of heap. Try setting to 128k or 64k(min) to see if that will cause less GCS and help with performance.

Solaris-Only flags

• -XX:TLESize
  Size of the thread local portions of heap in young generation only in the server vm.
• -XX:-UseTLE
  Turn off thread local heaps.
  For the server vm, you could completely eliminate TLE or tune the size so that they don't impact heap space when there are many threads.
• -XX:+OverrideDefaultLibthread
• -XX:+UseLWPSynchronization
  
  • If you find that EAServer is causing a lot of context switches, or observe an abnormally high amount of %sys cpu usage on a multi-cpu Solaris machine when running EAServer, try using the alternative thread-library that comes with Solaris 2.8 and turning on the -XX:+OverrideDefaultLibthread

    In order to use the alternative thread library, follow these steps:

    1. add -XX:+OverrideDefaultLibthread to JVM options
    2. in shell: LD_LIBRARY_PATH=/usr/lib/lwp
       export LD_LIBRARY_PATH
    3. start EAServer
  • On Solaris 2.6, try using -XX:+UseLWPSynchronization to see if that helps as the alternative thread library is not available in 2.6
• -XX+UseISM
  Intimate Shared Memory (ISM). ISM provides the benefit of having larger pages(4mb) and locking pages to memory. If you have enough memory to dedicate to EAServer, then turning on ISM could help performance. Be aware that ISM does not guarantee larger pages unless your system has been rebooted and there is enough contiguous physical memory to fulfill ISM needs.

  Requirements in /etc/system
  

  • 2.8:
    · set shmsys:shminfo_shmmax=0xffffffff
    · set shmsys:shminfo_shmseg=32
    
  • 2.6:
    · set shmsys:shminfo_shmmax=0xffffffff
    · set shmsys:shminfo_shmseg=32
    · set enable_grp_ism=1
• -XX:-DisableExplicitGC
  Disable calls to explicit GC.
  

  EAServer calls System.gc() periodically. You can try disabling this to see if it will help GC performance. The System.gc called is just a hint, so turning on DisableExplicitGC might not do anything.

Allow Just-In-Time compiling (JIT) by setting the com.sybase.jaguar.server.jvm.nojit property to false. JIT converts Java bytecode into native machine code, which generally runs much faster than when the bytecode is executed by the interpreter.

"Repository Properties Reference" in EAServer System Administration Guide

java.sun.com/developer/onlineTraining/Programming/JDCBook/perf2.html#jit">Just-In-Time Compilers on Java Developer Connection

For improved performance, EAServer 4.0+ can run with JDK 1.3 and the Java HotSpot virtual machine. On some platforms, EAServer can also use the Java HotSpot VM with JDK 1.2.

"Creating and Configuring Servers" in EAServer System Administration Guide

java.sun.com/products/hotspot/whitepaper.html">Java HotSpot Performance Engine Architecture

Be sure all unnecessary tracing is off. This includes the server-level trace options for Attentions, Network Driver APIs, Network Driver Requests, Protocol Data, Protocol Headers, Servlets, JAAS Debug, JCM Trace.

EAServer 4.1+ supports caching of static content. Static page caching can increase performance by caching static files in memory. You can configure the cache size, which files are cached, and how long the cached content is stored.

• Cache Size
  The default cache size is 10 megabytes, which should be sufficient for most uses unless an application includes a lot of static images/texts.
• Cache timeout
  The default timeout is 10 minutes. Set this to a higher value when you know the static content being served does not change often.
• Exclusion List
  Exclude-files are useful for turning off things that you know are used sparingly to ensure the cache stays relevant. You wouldn't want to inadvertently bump off 100 cached images because a request came in for a 5mb PDF that is only used once. See:
  • com.sybase.jaguar.server.http.cache.exclude-files
  • com.sybase.jaguar.server.http.cache.webapps.exclude-files

You can set the intervals for metrics collection and distribution;

Performance decreases as the frequency setting is set shorter, since it causes servers to ping each other more frequently. Ensure Heartbeat Frequency is set reasonably.

If the cluster includes components that enable object caching automatic persistence, you can choose one of four synchronization options for the caches; None (default), Mirror, Replicate, Invalidate.

The Replicate and Invalidate options provide two alternative ways to replicate with transaction consistency, if you need to do so. Between the two, the Invalidate option may yield better performance if the component's state is large, and the cluster has many members.

Object instantiation is expensive. To improve overall performance, use instance pooling, which allows EAServer clients to reuse component instances, eliminating the resource drain caused by repeated creation of component instances.

Instance pooling can decrease client response time, but can also increase memory usage in the Jaguar server. You can configure the maximum and minimum pool size to constrain the memory used to maintain an instance pool. By default, no minimum or maximum is set. You can increase memory efficiency by ensuring that heavily used components have higher minimum and maximum pool sizes than less commonly used components.

If you specify a maximum, the pool size will grow as additional instances are required to satisfy client requests, up to the maximum specified size. Subsequently, EAServer destroys excess instances after deactivation. As a result, instance pooling benefits are lost when the maximum is configured too low.

Named instance pools are available in EAServer 3.6.1 ESD4, and 4.1.1+. This feature allows specific components to be grouped into specific pools, with minimum and maximum instances defined.

This feature can be especially useful for components that have the Bind Thread property set.

When this option is enabled, component instances are bound to the creating thread. Although components with this setting may be pooled, and multiple instances may still run concurrently on separate threads, scalability is decreased. In addition, extra resources are required since twice as many threads are needed -- the client thread plus another thread bound to the component.

Enable this option only for ActiveX components, and components that use thread-local storage (see the PowerBuilder components section). Otherwise, disable this feature so EAServer can run the component on any available thread.

Avoid this model - it uses stateful components and does not use pooling, which greatly reduces scalability.

Bind object causes an instance of a stateful component to be bound to a client's proxy reference until the client destroys or releases the reference, or the client times out.

Concurrency indicates whether multiple instances can be simultaneously active on different threads. Set to true by default, to offer the highest performance.

Important: With concurrency set, you must protect any static instance variables by making them thread-safe. Exceptions to this rule are static variables that include the final modifier (meaning it is a constant that cannot be changed), and static variables defined as a primitive datatype that is 32 bits or less. This restriction is required in order to prevent thread race conditions, which cause random unpredictable results that can be very difficult to troubleshoot.

"Component properties: Instances" in EAServer Programmer's Guide

java.sun.com/developer/Books/performance2/chap3.pdf">Race Conditions and Mutual Exclusion on the Sun java.sun.com/developer/Books/performance2/">Java Developer Connection

Avoid this model. Since there is one instance per server, you must synchronize all instance and static variables, which creates a bottleneck each time a synchronization is done.

Additionally, developers need to be aware that in a cluster, each server has it's own instance, so the component is not truly a singleton object.

The shared flag should only be used in specific scenarios where the component is designed with the above considerations in mind, and the developer is aware of the potential server bottlenecks and cluster implications.

1. Use connection caching and database commands to accumulate data in a remote database. This is the preferred technique.
2. Accumulate data in the client using a data structure ( IDL structure and sequence types) that is passed to each method invocation. This technique is only practical if the amount of data is small. Sending large amounts of data over the network will degrade performance.
3. If the accumulated data is small and represented by simple data structures, you can store the data in a local file.
4. For ActiveX, C, or C++ components, use the EAServer shared objects interface, which allows components to store references to shared data.

State data can be stored either in memory or to a persistent data store. In-memory storage offers better performance than persistent storage.

In-memory storage uses a mirror-pair model where data is replicated between pairs of servers running in the cluster. Each client session has two points of failure (the originating server, and its mirror-pair twin.)

Persistent storage uses a remote database to store component state. A component instance can failover to any other server in the cluster where the component is installed. Persistent storage requires a highly available database, otherwise the database itself can become a single point of failure.

1. Code your component to call the completeWork or rollbackWork transaction state primitive to cause explicit deactivation of the instance. This technique is useful when your design requires deactivation to occur after some, but not all, method calls.
2. Enable the Auto demarcation/deactivation property for the component, so the component is automatically deactivated after every method invocation.

Be sure all tracing and debugging flags are turned off in the production environment. These properties include:

• com.sybase.jaguar.component.debug
• com.sybase.jaguar.component.cpp.debug
• com.sybase.jaguar.component.pb.debug
• com.sybase.jaguar.component.trace

By default, EAServer will wait indefinitely for component methods to complete, for instances to be released, and for open transactions to be completed.

If desired, you can set a maximum number of seconds that EAServer should wait for these events. You can set defaults at the server-level, and override the defaults at the component-level. When a specified maximum is reached, EAServer will returning an exception to the client.

(All of the above properties are prefixed by com.sybase.jaguar.)

(*) Note: Network latency between client and server is not included in the measured method execution time. Although for C++ components running in an external process, the measured time does include interprocess communication latency.

For best performance, enable the Concurrency option and disable the Bind Thread option.

1. Concurrency allows multiple method invocations to occur simultaneously, which can decrease client response time. Also, if your component has a run() method that executes indefinitely, you must enable the Concurrency option or no clients will be able to invoke methods.
2. Disabling the Bind Thread option allows EAServer to run the component on any available thread. (This option should only be enabled for ActiveX components and components that use thread-local storage.)

Several profiling tools are available to help identify bottlenecks, memory leaks, and other performance issues in a Java components, applications, and applets. Profiling can help you identify code that needs to be modified or removed for performance reasons.

Setting up OptimizeIt for Use with EAServer

Memory Management within Java Processes

Java 2 Performance Analysis on Java Developer Connection