1. 问题描述

XXX的wls816在搭建多数据源测试环境，多数据源策略使用High-Availability，正常情况下当datasource1出现问题不可用时，datasource2接管应用数据库连接请求，应用使用不受影响，当datasource1恢复后，应用连接重新回到datasource1上，该过程对客户端透明。使用loadrunner进行压测过程中发现，当停掉datasource1对应的数据库实例1时，应用使用异常缓慢，datasource1状态变为unhealth，连接转到datasource2上，当恢复datasource1对应的数据库实例1时连接可以重新回到datasource1上。

2. 应用环境

App服务器：WebLogic Server 8.1 SP6

Linux
BEA JRockit 1.4.2_10
DomainName 7011domain

3. 问题分析

当停掉datasource1对应的数据库实例1后，weblogic后台开始大量报如下错误：

####<2013-5-2 下午03时17分17秒 CST> <Error> <JDBC> <press01> <myserver> <ExecuteThread: '197' for queue: 'weblogic.kernel.Default'> <<anonymous>> <> <BEA-001255> <MultiPool gpicConnectionPool unable to disable connection pool testDB1, got exception: weblogic.common.resourcepool.ResourcePermissionsException: User "<anonymous>" does not have permission to perform operation "admin" on resource "testDB1" of application "null" of type "ConnectionPool".>

由于weblogic不能把datasource1 disable掉，所以weblogic会一直测试连接池内的所有数据库连接并重建，由于当时数据库实例1已经不可用，所以再次创建连接依然不成功，最后所有连接测试和重建连接完成后，weblogic才把应用的请求交给datasource2处理，从而大大的影响了系统性能。通过查找oracle weblogic官方文档，发现高级参数CountOfRefreshFailuresTillDisable可以规避该问题，但该参数不能通过管理控制台设置，需要在jdbc配置文件中设置。官方解释如下：

Minimizing Connection Request Delay After Connection Test Failures
If your DBMS becomes and remains unavailable, the connection pool will persistently test and try to replace dead connections while trying to satisfy connection requests. This behavior is beneficial because it enables the connection pool to react immediately when the database becomes available. However, testing a dead database connection can take as long as the network timeout, and can cause a long delay for clients.
To minimize the delay that occurs for client applications while a database is unavailable, you can set the CountOfRefreshFailuresTillDisable attribute on the connection pool. With this attribute set, WebLogic Server disables the connection pool after the number of consecutive failures to replace a dead connection. When an application requests a connection from a disabled connection pool, WebLogic Server throws a ConnectDisabledException immediately to notify the client that a connection is not available.
For connection pools that are disabled in this manner, WebLogic Server periodically run the refresh process. When the refresh process succeeds in creating a new database connection, WebLogic Server re-enables the connection pool. You can also manually re-enable the connection pool using the administration console or the weblogic.Admin ENABLE_POOL command.
You specify the CountOfRefreshFailuresTillDisable attribute in the JDBCConnectionPool entry in the config.xml file. TestConnectionsOnReserve must also be set to true. For example:
<JDBCConnectionPool
CapacityIncrement="1"
DriverName="com.pointbase.xa.xaDataSource"
InitialCapacity="2" MaxCapacity="10"
Name="demoXAPool" Password="password"
Properties="user=examples;
DatabaseName=jdbc:pointbase:server://localhost/demo"
Targets="examplesServer"
TestConnectionsOnReserve="true"
CountOfRefreshFailuresTillDisable="1"
TestTableName="SYSTABLES"
URL="jdbc:pointbase:server://localhost/demo"
/>
Note: The CountOfRefreshFailuresTillDisable attribute is not available in the Administration Console.
If you tend to see small network glitches or have a firewall that may occasionally kill only one socket or connection, you may want to set the number of refresh failures to 2 or 3, but a value of 1 will usually provide the best performance.

通过该参数设置可以使当第一次连接测试失败后就把数据库连接池disable掉，从而weblogic多数据源可以直接把应用请求发送到datasource2上的连接池去处理，而不用测试池内所有的连接节省了大量的时间。一般官方建议该参数配合使用CountOfTestFailuresTillFlush参数配合使用，该参数代表当测试连接失败n次后把整个池内的连接都关闭掉。

设置上述两个参数的值都为-1后，重新测试，当停掉数据库实例1后，系统运行正常，weblogic后台没有报错，连接全部转到datasource2的连接池内，恢复数据库实例1后，连接自动重连到datasource1的连接池内，通过几次测试系统运行均正常