Bugzilla – Bug 4110
HLDS Linux CS1.6 keeps crashing 1-8 times/day, debug info provided
Last modified: 2013-06-15 15:02:06 PDT
Created attachment 1816 [details] Contains the bt output from the core dumps and the server setup. Hello, this is my first report so forgive me if anything is done incorrect (like the component selected above). We have been experiencing a lot of various crashes, mainly segmentation faults, on our 10 public servers since we made the switch from amxx 1.76d to amxx 1.8.1.3746 (never tried 1.8.0). Previously I have written the errors off as plugin errors since we updated our mod servers first (gungame, wc3ft etc), as mentioned in this thread (https://forums.alliedmods.net/showthread.php?t=75312 ). But this report is based solely on our simplest server; a standard cs 1.6 hlds where only dust2 is played, without any big gameplay modifier like wc3/dm. This means it has far less plugins running than the other servers, but still it crashes almost daily, up to many times a day. We never had crashes more often than once every ~2 weeks on amxx 1.76d, but on 1.8 there are servers crashing very often. To battle this problem we have installed gdb and are running in debug mode to find issues, and we make hlds restart auto on crash. We are still at a loss as to what causes these problems, and I am posting here hoping for someone to shed some light or find something that can be fixed/altered to secure, more stabile servers for us. This is a 1U server with a Supermicro MB and specs: CPU: Intel Core 2 Duo E6600 @2,4Ghz RAM: 2GB DDR2 HDD: 74GB WD Raptor 10000rpm The server load reported by Munin peaks at max 0.1 each day, so the HW is probably not overloaded in any way. This server is on a 1Gbit bandwith with a couple of other servers and should have plentiful BW available. Please see the attachment for explanation of the server setup, as well as the output from running bt commands on all the core dumps I've saved up. The actual core.* dumps are around 110MB in size, but I will be happy to serve them up if you want them. I hope you can take a look at these and I would love if you could help me stop the servers from crashing or at least lower the amount of crashes per day.
On the server that has the most reduced subset of AMX Mod X related stuff... Please post the output of the following commands from the server console: meta list amxx plugins amxx moduls
Ah, sorry. Had that ready but forgot to include it. Data follows, I pasted it here as well in case this page doesn't preserve whitespace: http://pastie.org/699578 meta list Currently loaded plugins: description stat pend file vers src load unlod [ 1] AMX Mod X RUN - amxmodx_mm_i386. v1.8.1.3 ini Start ANY [ 2] MySQL RUN - mysql_amxx_i386. v1.8.1.3 pl2 ANY ANY [ 3] Fun RUN - fun_amxx_i386.so v1.8.1.3 pl2 ANY ANY [ 4] Engine RUN - engine_amxx_i386 v1.8.1.3 pl2 ANY ANY [ 5] FakeMeta RUN - fakemeta_amxx_i3 v1.8.1.3 pl2 ANY ANY [ 6] CStrike RUN - cstrike_amxx_i38 v1.8.1.3 pl2 ANY ANY [ 7] CSX RUN - csx_amxx_i386.so v1.8.1.3 pl2 ANY ANY [ 8] Ham Sandwich RUN - hamsandwich_amxx v1.8.1.3 pl2 ANY ANY 8 plugins, 8 running amxx plugins Currently loaded plugins: name version author file status [ 1] Admin Commands 1.8.1.3746 AMXX Dev Team admincmd.amxx debug [ 2] Menus Front-End 1.8.1.3746 AMXX Dev Team menufront.amxx debug [ 3] Commands Menu 1.8.1.3746 AMXX Dev Team cmdmenu.amxx debug [ 4] Players Menu 1.8.1.3746 AMXX Dev Team plmenu.amxx debug [ 5] Admin Chat 1.8.1.3746 AMXX Dev Team adminchat.amxx debug [ 6] Anti Flood 1.8.1.3746 AMXX Dev Team antiflood.amxx debug [ 7] Info. Messages 1.8.1.3746 AMXX Dev Team imessage.amxx debug [ 8] Restrict Weapons 1.8.1.3746 AMXX Dev Team restmenu.amxx debug [ 9] StatsX 1.8.1.3746 AMXX Dev Team statsx.amxx debug [ 10] ColorChat 0.0.7 ConnorMcLeod colorchat.amxx debug [ 11] ag_member 1.3.1 [ag] crew ag_member.amxx debug [ 12] ag_connection 1.0 supergreg ag_connection.a debug [ 13] Slay/slap team 1.0 SGulseth ag_teampunish.a debug [ 14] AMXBans 5.1rc YoMama/Lux & lan amxbans.amxx debug [ 15] Admin ESP mini 1.6 KoST, supergreg admin_esp.amxx debug [ 16] Invisible Spectator 0.2 ConnorMcLeod invisible_spect debug [ 17] HLDS<->IRC 2.7 devicenull irc.amxx debug [ 18] Team Balancer 1.8b3 Ptahhotep ptb.amxx debug [ 19] Anti DoubleDuck 1.6 Numb antidoubleduck. debug [ 20] AFK to Spectator 1.0 bugsy afk-spec.amxx debug [ 21] Team Flash Punish 1.1.0 ConnorMcLeod teamflash_punis debug [ 22] Ultimate Chat 1.2 [ --<-@ ] ultimate_chat.a debug [ 23] GHW Say Commands 1.1 GHW_Chronic say_commands.am debug [ 24] Reconnect Features 0.2.4 BETA ConnorMcLeod reconnect_featu debug [ 25] Real Nade Drops 0.4 VEN realnadedrops.a debug 25 plugins, 25 running amxx modules Currently loaded modules: name version author status [ 1] MySQL 1.8.1.3746 AMX Mod X Dev Team running [ 2] Fun 1.8.1.3746 AMX Mod X Dev Team running [ 3] Engine 1.8.1.3746 AMX Mod X Dev Team running [ 4] FakeMeta 1.8.1.3746 AMX Mod X Dev Team running [ 5] nVault 1.8.1.3746 AMX Mod X Dev Team running [ 6] CStrike 1.8.1.3746 AMX Mod X Dev Team running [ 7] CSX 1.8.1.3746 AMX Mod X Dev Team running [ 8] Ham Sandwich 1.8.1.3746 AMX Mod X Dev Team running [ 9] Sockets 1.8.1.3746 HLSW Dev Team running [10] RegEx 1.8.1.3746 AMX Mod X Dev Team running 10 modules, 10 correct
Try to disable all plugins exepts defaults and test one per one. To see how is crashing you server. Or put debug near the plugin name and then look the logs!!
All plugins have been running in debug mode for several months, that's how I was able to generate the error reports in the first post here. You can also see this in comment #2. I will however try to run an even more limited set of plugins starting today. Will report back when I am done testing all these plugins one by one, should be in a couple of weeks.
Hello again David, we have now ran this server with an even more limited set of plugins, and yet seem to get the broken pipe crash a couple of times a week. zipfile: http://dump.no/files/2983bff81719/coredumps-v5.zip contents: config-v2.txt - Ouput of amxx plugins/modules for this entire period gdb.txt - Entire gdb.txt log for period core.*.txt - Output of bt and info thread on the dumps As you may notice gdb.txt mentions a lot more dumps than the ones attached. It seems when it crashes once, it generates 1-4 dumps before it gets back on its feet, so I have only included the first one for each crash. I hope this aids in getting to the bottom of the issue, some of our admins are not pleased with having plugins taken away from them, hehe. Regardless, if you have any request for specific things we could try, please ask. I'll happily jump through hoops if the end result is more stable servers.
Down to only 3 third party plugins, and still getting segfaults stating "broken pipe". These 3rd party are: Amxbans 5.1rc, PTB, and IRC (logs normal chat to irc), all newest available versions. These same plugins are also in use on our amxx 1.76d servers without a hitch, even together with a bunch of other plugins. I see the link in my previous comment is already dead, so I put both the old v5 and the current v6 info here: http://pegs.no/coredumps/