Bug 4110 - HLDS Linux CS1.6 keeps crashing 1-8 times/day, debug info provided
HLDS Linux CS1.6 keeps crashing 1-8 times/day, debug info provided
Status: RESOLVED INCOMPLETE
Product: AMX Mod X
Classification: Unclassified
Component: Core
trunk
PC Linux
: P3 normal
Assigned To: amxmodx-bugs@alliedmods.net
:
Depends on:
Blocks:
  Show dependency treegraph
 
Reported: 2009-11-15 00:31 PST by Chris
Modified: 2013-06-15 15:02 PDT (History)
4 users (show)

See Also:


Attachments
Contains the bt output from the core dumps and the server setup. (67.74 KB, application/x-zip-compressed)
2009-11-15 00:31 PST, Chris
no flags Details

Description Chris 2009-11-15 00:31:14 PST
Created attachment 1816 [details]
Contains the bt output from the core dumps and the server setup.

Hello, this is my first report so forgive me if anything is done incorrect (like the component selected above).

We have been experiencing a lot of various crashes, mainly segmentation faults, on our 10 public servers since we made the switch from amxx 1.76d to amxx 1.8.1.3746 (never tried 1.8.0). Previously I have written the errors off as plugin errors since we updated our mod servers first (gungame, wc3ft etc), as mentioned in this thread (https://forums.alliedmods.net/showthread.php?t=75312 ).

But this report is based solely on our simplest server; a standard cs 1.6 hlds where only dust2 is played, without any big gameplay modifier like wc3/dm. This means it has far less plugins running than the other servers, but still it crashes almost daily, up to many times a day. We never had crashes more often than once every ~2 weeks on amxx 1.76d, but on 1.8 there are servers crashing very often.

To battle this problem we have installed gdb and are running in debug mode to find issues, and we make hlds restart auto on crash. We are still at a loss as to what causes these problems, and I am posting here hoping for someone to shed some light or find something that can be fixed/altered to secure, more stabile servers for us.

This is a 1U server with a Supermicro MB and specs:
CPU: Intel Core 2 Duo E6600 @2,4Ghz
RAM: 2GB DDR2
HDD: 74GB WD Raptor 10000rpm

The server load reported by Munin peaks at max 0.1 each day, so the HW is probably not overloaded in any way. This server is on a 1Gbit bandwith with a couple of other servers and should have plentiful BW available.

Please see the attachment for explanation of the server setup, as well as the output from running bt commands on all the core dumps I've saved up. The actual core.* dumps are around 110MB in size, but I will be happy to serve them up if you want them.

I hope you can take a look at these and I would love if you could help me stop the servers from crashing or at least lower the amount of crashes per day.
Comment 1 David Anderson [:dvander] 2009-11-15 06:34:21 PST
On the server that has the most reduced subset of AMX Mod X related stuff...

Please post the output of the following commands from the server console:

meta list
amxx plugins
amxx moduls
Comment 2 Chris 2009-11-15 07:43:19 PST
Ah, sorry. Had that ready but forgot to include it.
Data follows, I pasted it here as well in case this page doesn't preserve whitespace: http://pastie.org/699578

meta list
Currently loaded plugins:
      description      stat pend  file              vers      src   load  unlod
 [ 1] AMX Mod X        RUN   -    amxmodx_mm_i386.  v1.8.1.3  ini   Start ANY
 [ 2] MySQL            RUN   -    mysql_amxx_i386.  v1.8.1.3  pl2   ANY   ANY
 [ 3] Fun              RUN   -    fun_amxx_i386.so  v1.8.1.3  pl2   ANY   ANY
 [ 4] Engine           RUN   -    engine_amxx_i386  v1.8.1.3  pl2   ANY   ANY
 [ 5] FakeMeta         RUN   -    fakemeta_amxx_i3  v1.8.1.3  pl2   ANY   ANY
 [ 6] CStrike          RUN   -    cstrike_amxx_i38  v1.8.1.3  pl2   ANY   ANY
 [ 7] CSX              RUN   -    csx_amxx_i386.so  v1.8.1.3  pl2   ANY   ANY
 [ 8] Ham Sandwich     RUN   -    hamsandwich_amxx  v1.8.1.3  pl2   ANY   ANY
8 plugins, 8 running

amxx plugins
Currently loaded plugins:
       name                    version     author            file             status
 [  1] Admin Commands          1.8.1.3746  AMXX Dev Team     admincmd.amxx    debug
 [  2] Menus Front-End         1.8.1.3746  AMXX Dev Team     menufront.amxx   debug
 [  3] Commands Menu           1.8.1.3746  AMXX Dev Team     cmdmenu.amxx     debug
 [  4] Players Menu            1.8.1.3746  AMXX Dev Team     plmenu.amxx      debug
 [  5] Admin Chat              1.8.1.3746  AMXX Dev Team     adminchat.amxx   debug
 [  6] Anti Flood              1.8.1.3746  AMXX Dev Team     antiflood.amxx   debug
 [  7] Info. Messages          1.8.1.3746  AMXX Dev Team     imessage.amxx    debug
 [  8] Restrict Weapons        1.8.1.3746  AMXX Dev Team     restmenu.amxx    debug
 [  9] StatsX                  1.8.1.3746  AMXX Dev Team     statsx.amxx      debug
 [ 10] ColorChat               0.0.7       ConnorMcLeod      colorchat.amxx   debug
 [ 11] ag_member               1.3.1       [ag] crew         ag_member.amxx   debug
 [ 12] ag_connection           1.0         supergreg         ag_connection.a  debug
 [ 13] Slay/slap team          1.0         SGulseth          ag_teampunish.a  debug
 [ 14] AMXBans                 5.1rc       YoMama/Lux & lan  amxbans.amxx     debug
 [ 15] Admin ESP mini          1.6         KoST, supergreg   admin_esp.amxx   debug
 [ 16] Invisible Spectator     0.2         ConnorMcLeod      invisible_spect  debug
 [ 17] HLDS<->IRC              2.7         devicenull        irc.amxx         debug
 [ 18] Team Balancer           1.8b3       Ptahhotep         ptb.amxx         debug
 [ 19] Anti DoubleDuck         1.6         Numb              antidoubleduck.  debug
 [ 20] AFK to Spectator        1.0         bugsy             afk-spec.amxx    debug
 [ 21] Team Flash Punish       1.1.0       ConnorMcLeod      teamflash_punis  debug
 [ 22] Ultimate Chat           1.2         [ --<-@ ]         ultimate_chat.a  debug
 [ 23] GHW Say Commands        1.1         GHW_Chronic       say_commands.am  debug
 [ 24] Reconnect Features      0.2.4 BETA  ConnorMcLeod      reconnect_featu  debug
 [ 25] Real Nade Drops         0.4         VEN               realnadedrops.a  debug
25 plugins, 25 running

amxx modules
Currently loaded modules:
      name                    version     author               status
 [ 1] MySQL                   1.8.1.3746  AMX Mod X Dev Team   running
 [ 2] Fun                     1.8.1.3746  AMX Mod X Dev Team   running
 [ 3] Engine                  1.8.1.3746  AMX Mod X Dev Team   running
 [ 4] FakeMeta                1.8.1.3746  AMX Mod X Dev Team   running
 [ 5] nVault                  1.8.1.3746  AMX Mod X Dev Team   running
 [ 6] CStrike                 1.8.1.3746  AMX Mod X Dev Team   running
 [ 7] CSX                     1.8.1.3746  AMX Mod X Dev Team   running
 [ 8] Ham Sandwich            1.8.1.3746  AMX Mod X Dev Team   running
 [ 9] Sockets                 1.8.1.3746  HLSW Dev Team        running
 [10] RegEx                   1.8.1.3746  AMX Mod X Dev Team   running
10 modules, 10 correct
Comment 3 Ramón 2009-11-23 12:06:20 PST
Try to disable all plugins exepts defaults and test one per one. To see how is crashing you server. Or put debug near the plugin name and then look the logs!!
Comment 4 Chris 2009-11-23 22:37:58 PST
All plugins have been running in debug mode for several months, that's how I was able to generate the error reports in the first post here. You can also see this in comment #2.

I will however try to run an even more limited set of plugins starting today. Will report back when I am done testing all these plugins one by one, should be in a couple of weeks.
Comment 5 Chris 2009-12-06 11:49:23 PST
Hello again David, we have now ran this server with an even more limited set of plugins, and yet seem to get the broken pipe crash a couple of times a week.

zipfile: http://dump.no/files/2983bff81719/coredumps-v5.zip
contents:
 config-v2.txt - Ouput of amxx plugins/modules for this entire period
 gdb.txt - Entire gdb.txt log for period
 core.*.txt - Output of bt and info thread on the dumps

As you may notice gdb.txt mentions a lot more dumps than the ones attached. It seems when it crashes once, it generates 1-4 dumps before it gets back on its feet, so I have only included the first one for each crash.

I hope this aids in getting to the bottom of the issue, some of our admins are not pleased with having plugins taken away from them, hehe. Regardless, if you have any request for specific things we could try, please ask. I'll happily jump through hoops if the end result is more stable servers.
Comment 6 Chris 2009-12-12 12:25:57 PST
Down to only 3 third party plugins, and still getting segfaults stating "broken pipe". These 3rd party are: Amxbans 5.1rc, PTB, and IRC (logs normal chat to irc), all newest available versions.
These same plugins are also in use on our amxx 1.76d servers without a hitch, even together with a bunch of other plugins. 

I see the link in my previous comment is already dead, so I put both the old v5 and the current v6 info here: http://pegs.no/coredumps/

Note You need to log in before you can comment on or make changes to this bug.