Purpose

The purpose of this page is to discuss failover and load balancing considerations for MobiLink Synchronization server farms.

Introduction

A server farm is used to provide load balancing and failover for mission critical applications. The MobiLink server can run in this configuration, but there are a few considerations to bear in mind when dealing with concurrency and failed synchronizations. This document describes how to set up a MobiLink server farm to handle these scenarios.

MobiLink in a Failover and Load Balancing Environment

The following diagram illustrates a potential MobiLink configuration in a failover and load balancing environment.

MobiLink Requirements for Failover

MobiLink servers are stateless, in that they do not remember any state information between synchronization sessions. State information for each remote ID is stored in the MobiLink system tables, which are automatically maintained in the consolidated database. When a MobiLink server goes down, all current synchronization on that MobiLink server stops and returns as failed, but these failures are only temporary and the system automatically recovers during the next synchronization. On the next synchronization, the MobiLink server ensures that all data is properly synchronized, provided there are no other issues causing failures.

When a MobiLink server fails, there must be a load balancer to perform the following operations:

1. Detect whether the MobiLink server is available.
2. Send synchronizations to the MobiLink server when it becomes available.
3. Stop sending synchronizations to the MobiLink server when it is unavailable.
4. Prevent concurrent synchronization from the same remote ID.

MobiLink Requirements for Load Balancing

Before discussing the requirements for running the MobiLink server in a load balancing environment, it is important to highlight the challenges faced by a MobiLink system in general.

Concurrent Synchronizations

For each remote ID that synchronizes, the MobiLink server stores state information in MobiLink system tables. The state information is both queried and changed at different points during synchronization. These changes occur within the multiple transactions that make up a complete synchronization session. If two different MobiLink servers concurrently synchronized the same remote ID, a conflict would arise as they interleaved two different sets of updates to the same remote ID state. For this reason, MobiLink architecture forbids concurrent synchronizations from the same remote ID.

Within a single MobiLink server, there is a built-in mechanism for preventing concurrent synchronizations from the same remote ID. In SQL Anywhere 10.0.1, there is no built-in mechanism for preventing two or more MobiLink servers synchronizing the same remote ID concurrently. Thus, a load balancer is required to solve this problem in the MobiLink synchronization environment.

Client/ Server Communication Failures are Common

Network connectivity can be sporadic, particularly over public wireless, dial-up, and even some ISPs. A communication failure can easily cause concurrent synchronizations using the same remote ID, as in the following example:

1. Remote ID R1 is in the middle of a long synchronization.
2. A communication failure occurs. Note that the MobiLink server is still processing the synchronization and may not notice the failure yet, either because of network conditions or because the synchronization is performing a lengthy operation in the consolidated database at the time.
3. The user of R1 sees the failure and retries the synchronization immediately.
4. The second synchronization begins processing in a second MobiLink server, while the first synchronization is still in progress in the first MobiLink server. The second synchronization is concurrent with the first one.

Note that even a successful first synchronization may still be processing on the server after the remote has been notified that the synchronization is complete. This can occur when the same remote quickly performs several synchronizations in succession. A load balancer prevents concurrent synchronizations that use the same remote ID.

What a Load Balancer Can Do to Prevent Concurrent Synchronization

A simple way for a load balancer to prevent concurrent synchronizations using the same remote ID is to take advantage of the MobiLink server’s ability to detect the problem within a single server instance. If the load balancer always sends the same remote ID to the same MobiLink server, then that MobiLink server could reliably prevent concurrent synchronizations of that remote ID.

Many load balancers support the concept of stickiness, or persistence. This feature works as follow:

• Each time an IP address connects to the load balancer, the load balancer looks for an existing association between an IP address and a MobiLink server. If an association exists, the load balancer sends the connection to the same MobiLink server; otherwise, it picks a new MobiLink server, records the association, and forwards the connection (this is the stickiness/persistence part).
• When an IP address connects to the load balancer for the first time, the load balancer randomly (or based on load) picks a MobiLink server to handle the connection. The load balancer records the association between the IP address and the specific MobiLink server. Removing stickiness/persistence lets the same IP address get load balanced to a different MobiLink server later. This should help reduce the skewed loads sometimes caused by stickiness/persistence.

The load balancer’s timeout must be set longer than the longest possible synchronization duration. Otherwise, it is possible for concurrent synchronizations of the same remote ID to occur. This duration can be difficult to determine, so a sensible choice is a duration that is guaranteed to be much longer than a single synchronization. Benchmark your synchronizations to determine the maximum amount of time for a single synchronization.

When Stickiness/ Persistence is Not Feasible

Using stickiness/persistence requires that all remote IDs consistently use the same IP address. Unfortunately, some environments such as public wireless, rarely, if ever, allow static IP assignments. How can concurrent synchronizations be prevented when using IP addresses that are dynamic or even volatile?

TCP/IP Synchronization

When synchronizing with TCP/IP (including TLS), there is nothing the load balancer can do. Without internal knowledge of the MobiLink protocol or a reliable IP address, it cannot behave as required. The only solution in this case is to prevent concurrent synchronizations by never synchronizing the same remote ID more than once during a period of time equal to the longest possible synchronization, plus the TCP/IP session timeout value. If the MobiLink synchronization client specifies a timeout value, then it is used; otherwise, the MobiLink server defaults to 240 seconds for the timeout value.

Consider two simple examples. In the first example, as above, immediately synchronizing after a communication failure can lead to concurrent synchronization of the same remote ID. In the second example, if the second synchronization is always delayed by at least a day, it is extremely unlikely that the problem will occur. The long (adequate) delay between synchronizations easily prevents the problem.

HTTP/HTTPS Synchronization

When synchronizing with HTTP or HTTPS, there is a solution: use a smart load balancer that can route traffic using custom rules based on HTTP headers.

The MobiLink HTTP/HTTPS implementation includes custom HTTP headers containing session information such as the remote ID. The format of the header is as follows:

ml-client-id: remote ID

The load balancer must perform the following operations:

• Inspect the HTTP headers. The load balancer looks for the ml-client-id header and extracts the remote ID.
• Each time a remote ID connects, the load balancer looks for an existing association between a remote ID and a MobiLink server. If an association exists, the load balancer sends the connection to the same MobiLink server; otherwise, it picks a new MobiLink server, records the association, and forwards the connection.
• When a remote ID connects for the first time, the load balancer randomly (or based on load) picks a MobiLink server to handle the connection. The load balancer records the association between the remote ID and the specific MobiLink server. Removing stickiness/persistence lets the same remote ID get load balanced to a different MobiLink server later. This should help reduce the skewed loads sometimes caused by stickiness/persistence

Note that HTTP-based load balancing is required when multiple Redirectors are used because each Redirector separately keeps an association between a remote ID and a MobiLink user. However, there is no coordination between Redirectors, and they might redirect the same remote ID to different MobiLink servers, causing concurrent synchronizations of the same remote ID.

A new feature was added to the MobiLink Server in v17.0.10 build 6073 in an attempt to allow the MobiLink Server to work with more restrictive load balancers, or with load balancers outside of your control.  If your load balancer cannot be configured to inspect the ml-client-id custom HTTP header, but can be configured to inspect HTTP cookies to perform load balancing, you can now use the client_id_cookie stream option with the HTTP and HTTPS streams to cause the MobiLink Server to also set an HTTP cookie with the given name that contains the same value placed in the ml-client-id custom HTTP header. For example, the MobiLink Server startup options below will set the commonly used JSESSIONID cookie with the value of the ml-client-id header.

• -x http(port=8080;client_id_cookie=JSESSIONID)
• -x https(port=8443;client_id_cookie=JSESSIONID;identity=ml.crt;identity_password=never_publish_the_identity_password)

Summary

MobiLink server farms do not have a built-in method to handle the problem of concurrent synchronizations using the same remote ID. This issue is easily triggered by failed client/server network communications, so MobiLink deployments of server farms must be set up to avoid the problem through the use of a load balancer

The load balancer must perform the following operations:

1. Detect whether or not the MobiLink server is available.
2. Send synchronizations to the MobiLink server when it is available.
3. Stop sending synchronizations to the MobiLink server when it is unavailable.
4. Prevent concurrent synchronization from the same remote ID.

To load balance properly, while avoiding the problem of concurrent synchronizations of the same remote ID, the load balancer must be set up as follows:

Let time, T, in seconds, be significantly longer than the longest possible synchronization.

• If the remote IP addresses are both static and always visible to the load balancer:
  • Use a load balancer that supports both stickiness/persistence and a timeout on the stickiness/persistence, using the remote IP address.
• Otherwise:
• If using TCP/IP (including TLS), no load-balancer setup is required, but you must always wait at least T seconds before synchronizing again after a single synchronization, including successful synchronizations.
• If using HTTP or HTTPS, use a load balancer that supports both stickiness/persistence and a timeout on the stickiness/persistence, using the ml-client-id HTTP header.

Related Content

Related Documents

Related SAP Notes/KBAs