1.概述
对XXX、YYY的tuxapp1和tuxapp2进行内存使用情况监控，从全局可以看到内存使用率在逐渐升高，不到36小时内存使用率从45%增加到54%，而从单个进程进行了数据的收集和比对，下面简要介绍一下对比的结果。
2.数据收集及分析
2.1.收集方法
每小时打印出当前操作系统占用内存最大的前20个进程，进行数据对比：
上述内存均为缓慢式增加，但是发现有进程增加幅度较大，如下所示：
上述进程均有较大的内存增长，因此判断各个进程均有较小的内存增大，也有个别进程发生较大的内存占用，猜测为某一业务分析导致的突然性增加。
3.其他问题
在监控进程中发现个别进程长时间执行请求，与同组的其他服务的完成请求数有较大差异，下面对这些进程进行调查，执行时间长的进程表现为争用CPU，单个进程的CPU占用长期处于94%-100%。
查看系统调用发现在做如下操作：
( Attached to process 24988 ("qcscrm1l2server -C dom=ubss -g 300 -i 4021 -u crmapp1 -U /ngbss") [64-bit] )
[24988] In user-mode .................................................................. [running]
[24988] write(6, "\015\0\006\0\0\0\0\00305d0\0\0\0".., 21) ............................ = 21
[24988] read(6, "03c8\0\006\0\0\0\0\006\002\0\003".., 2064) ........................... = 968
[24988] In user-mode .................................................................. [running]
[24988] write(6, "\015\0\006\0\0\0\0\00305d1\0\0\0".., 21) ............................ = 21
[24988] read(6, "039f\0\006\0\0\0\0\006\002\0\003".., 2064) ........................... = 927
[24988] In user-mode .................................................................. [running]
[24988] write(6, "\015\0\006\0\0\0\0\00305d2\0\0\0".., 21) ............................ = 21
[24988] read(6, "039e\0\006\0\0\0\0\006\002\0\003".., 2064) ........................... = 926
[24988] In user-mode .................................................................. [running]
[24988] write(6, "\015\0\006\0\0\0\0\00305d3\0\0\0".., 21) ............................ = 21
[24988] read(6, "03ad\0\006\0\0\0\0\006\002\0\003".., 2064) ........................... = 941
[24988] In user-mode .................................................................. [running]
[24988] write(6, "\015\0\006\0\0\0\0\00305d4\0\0\0".., 21) ............................ = 21
[24988] read(6, "03ad\0\006\0\0\0\0\006\002\0\005".., 2064) ........................... = 941
[24988] In user-mode .................................................................. [running]
[24988] write(6, "\015\0\006\0\0\0\0\00305d5\0\0\0".., 21) ............................ = 21
[24988] read(6, "03ad\0\006\0\0\0\0\006\002\0\003".., 2064) ........................... = 941
[24988] In user-mode .................................................................. [running]
[24988] write(6, "\015\0\006\0\0\0\0\00305d6\0\0\0".., 21) ............................ = 21
[24988] read(6, "03ad\0\006\0\0\0\0\006\002\0\003".., 2064) ........................... = 941
[24988] In user-mode .................................................................. [running]
[24988] write(6, "\015\0\006\0\0\0\0\00305d7\0\0\0".., 21) ............................ = 21
[24988] read(6, "0390\0\006\0\0\0\0\006\002\0\003".., 2064) ........................... = 912
[24988] In user-mode .................................................................. [running]
[24988] write(6, "\015\0\006\0\0\0\0\00305d8\0\0\0".., 21) ............................ = 21
[24988] read(6, "0390\0\006\0\0\0\0\006\002\0\003".., 2064) ........................... = 912
[24988] In user-mode .................................................................. [running]
[24988] write(6, "\015\0\006\0\0\0\0\00305d9\0\0\0".., 21) ............................ = 21
[24988] read(6, "0391\0\006\0\0\0\0\006\002\0\003".., 2064) ........................... = 913
[24988] In user-mode .................................................................. [running]
[24988] write(6, "\015\0\006\0\0\0\0\00305da\0\0\0".., 21) ............................ = 21
[24988] read(6, "039f\0\006\0\0\0\0\006\002\0\003".., 2064) ........................... = 927
[24988] In user-mode .................................................................. [running]
[24988] write(6, "\015\0\006\0\0\0\0\00305db\0\0\0".., 21) ............................ = 21
[24988] read(6, "03bb\0\006\0\0\0\0\006\002\0\005".., 2064) ........................... = 955
而查看多个进程的系统调用发现Read和Write函数的第一个参数和第二个参数相同，猜测应该是对同一缓冲区进行操作。
查看堆栈如下：
-------------------------------- lwpid : 8078689 -------------------------------
0: c000000001472ea0 : __milli_memcpy() + 0xb80 (/bea/tuxedo/tuxedo10gR3/lib/libfml32.sl)
1: c0000000014436e0 : Fchg32() at Fchg32.c:541
2: c00000001106a880 : CFmlBuf::SetValue(unsigned int,void*,int,unsigned int)() at /opt/aCC/include_std/rw/string_ref:264
3: c00000001106c910 : CFmlBuf::SetString(char const*,std::basic_string<char,std::char_traits<char>,std::allocator<char> >,int)() at CFmlBuf.cpp:756
4: c000000012018e20 : long StaticDAO::jselsql<CFmlBuf>((%0=)CFmlBuf&,std::basic_string<char,std::char_traits<char>,std::allocator<char> > const&,int)() at StaticDAO.cpp:632
5: c0000000120184d0 : StaticDAO::jselsql(CFmlBuf&,std::basic_string<char,std::char_traits<char>,std::allocator<char> > const&,int)() at StaticDAO.cpp:675
6: c00000001945b400 : QueryVagueInfo::queryVagueCoreInfo(std::basic_string<char,std::char_traits<char>,std::allocator<char> > const&,databuf::DataBuf const&)() at queryVagueInfo.cpp:183
7: c000000019456760 : queryVagueInfo() at queryVagueInfo.cpp:41
8: c00000001b818ef0 : UniNodeSoProxyParser::parse(databuf::DataBuf const&)() at UniNodeParserFactory.cpp:57
9: c00000001b894ae0 : UniTradeMgr::ParseBody(databuf::DataBuf const&) const() at UniTradeMgr.cpp:340
10: c00000001b8695e0 : UniTradeMgr::UniGeneTrade(databuf::DataBuf&,databuf::DataBuf&)() at UniTradeMgr.cpp:173
11: c00000001b866ac0 : UniGeneTrade() at UniTradeMgr.cpp:137
12: c0000000111854a0 : DLActionHandle::execute(ebpm4c::Context&)() at lcuplugins.cpp:229
13: c0000000114b3840 : ebpm4c::Action::execute(ebpm4c::Context&)() at /ngbss/anta/lang/include/base/SmartPtr.h:110
14: c00000001149cf20 : ebpm4c::State::acceptToken(ebpm4c::Context&) const() at lcu.cpp:177
15: c0000000114b14b0 : ebpm4c::Transition::acceptToken(ebpm4c::Context&) const() at lcu.cpp:88
16: c00000001149d520 : ebpm4c::State::acceptToken(ebpm4c::Context&) const() at lcu.cpp:194
17: c0000000114b14b0 : ebpm4c::Transition::acceptToken(ebpm4c::Context&) const() at lcu.cpp:88
18: c0000000114a1590 : ebpm4c::StartState::acceptToken(ebpm4c::Context&) const() at lcu.cpp:217
19: c0000000114bcac0 : ebpm4c::ProcessDefinition::execute(ebpm4c::Context&) const() at lcu.cpp:325
20: c00000001127f2b0 : MWLCUImpl::execute(counted_ptr<CFmlBuf>)() at mwlcu.cpp:474
21: 4000000000013580 : MWSVC() + 0x430 (/ngbss/tuxapp1/bin/qcscrm1l2server)
22: c00000000c8829b0 : _tmsvcdsp() at tmsvcdsp.c:512
23: c00000000c8e0870 : _tmrunserver() at tmrunsvr.c:2118
24: c00000000c880890 : _tmstartserver() at tmstrtsrvr.c:168
25: 4000000000007d50 : main() at BS-255f.c:76
26: c000000000045900 : main_opd_entry() + 0x50 (/usr/lib/hpux64/dld.so)
请开发工程确认上述堆栈的作用并进行修正，这些进程还具有一些其他的相同点：都是qcscrm1l2队列的服务，从数据依赖路由应该是XXX的请求发送到该队列。大约在执行2-4W个请求后开始出现长时间执行的请求，而且该种进程无法被自动跳出，只能tmshutdown或者kill -9手动结束，考虑到这种情况建议对该服务添加SVCTIMEOUT参数，来使Tuxedo自动kill长时间执行的服务，提高系统的稳定性。
这种长时间运行出现的服务在5月16日升级完成到5月17日下午2点30分，每台机器出现4个，相当于每个qcscrm1l2队列出现2个，而配置的MIN个数为5个，而根据当时的运行情况来说，在业务高峰已经出现的服务的自增长来满足系统需要，如果长时间运行进程持续增长的话会导致队列内具有活力的进程减少，导致队列积压，从而导致超时请求增加。
该贴被funny编辑于2014-2-26 11:18:11该贴被funny编辑于2014-2-26 11:19:40该贴被funny编辑于2014-2-26 11:20:36
当前位置：
论坛首页 »
自由讨论区 »
MQ, Tuxedo及OLTP讨论区
»
发起投票
技术讨论
加好友
发消息
2014-2-26 11:15:34 |
赞(
操作
