MySQL Cluster problems during setup

[tdktank59]

So I am having some issues when trying to start my cluster up...

I have the following
1 Mgm node (192.168.129.23) (512mb Ram)
2 Data Nodes (192.168.182.46, 192.168.180.87) (1024mb Ram)
2 API Nodes (192.168.129.218, 192.168.180.6) (768mb Ram, also runs nginx/php)

My config for MGM is

[root@phoenix mysql-cluster]# cat /var/lib/mysql-cluster/config.ini 
[ndbd default]
# Options affecting ndbd processes on all data nodes:
NoOfReplicas=2    # Number of replicas
DataMemory=80M    # How much memory to allocate for data storage
IndexMemory=18M   # How much memory to allocate for index storage

[tcp default]

[ndb_mgmd]
hostname=192.168.129.23
datadir=/var/lib/mysql-cluster

[ndbd]
hostname=192.168.182.46
datadir=/usr/local/mysql/data

[ndbd]
hostname=192.168.180.87
datadir=/usr/local/mysql/data

[mysqld]
# SQL node options:
hostname=192.168.182.46

And each data node has this config:

[root@magneto ~]# cat /etc/my.cnf
[mysqld] 
ndbcluster
ndb-connectstring=192.168.129.23
default-storage-engine=NDBCLUSTER
 
#max_connections=341
#query_cache_size=16M
#thread_concurrency = 4
 
[mysql_cluster]
ndb-connectstring=192.168.129.23

When I start it up I get this from the data node

[root@magneto ~]# ndbd
2012-03-12 15:53:26 [ndbd] INFO     -- Angel connected to '192.168.129.23:1186'
2012-03-12 15:53:26 [ndbd] INFO     -- Angel allocated nodeid: 3

And on the MGM node I get this

[root@phoenix mysql-cluster]# ndb_mgm
-- NDB Cluster -- Management Client --
ndb_mgm> show
Connected to Management Server at: localhost:1186
Cluster Configuration
---------------------
[ndbd(NDB)] 2 node(s)
id=2 (not connected, accepting connect from 192.168.182.46)
id=3 @192.168.180.87 (mysql-5.5.19 ndb-7.2.4, starting, Nodegroup: 0)

[ndb_mgmd(MGM)] 1 node(s)
id=1 @192.168.129.23 (mysql-5.5.19 ndb-7.2.4)

[mysqld(API)] 2 node(s)
id=4 (not connected, accepting connect from any host)
id=5 (not connected, accepting connect from any host)

ndb_mgm> Node 3: Forced node shutdown completed. Occured during startphase 0. Initiated by signal 9.

Trying to figure out what is causing the forced shutdown but have yet to find anything, I have messed with memory settings and lowering them but that does not seem to help. Any ideas or places I should start looking?

Data Node log file

2012-03-12 16:01:15 [ndbd] INFO     -- Angel pid: 1766 started child: 1767
2012-03-12 16:01:15 [ndbd] INFO     -- Configuration fetched from '192.168.129.23:1186', generation: 1
NDBMT: non-mt
2012-03-12 16:01:15 [ndbd] INFO     -- NDB Cluster -- DB node 3
2012-03-12 16:01:15 [ndbd] INFO     -- mysql-5.5.19 ndb-7.2.4 --
2012-03-12 16:01:15 [ndbd] INFO     -- numa_set_interleave_mask(numa_all_nodes) : no numa support
2012-03-12 16:01:15 [ndbd] INFO     -- Ndbd_mem_manager::init(1) min: 1004Mb initial: 1132Mb
Adding 36Mb to ZONE_LO (1,1151)
Instantiating DBSPJ instanceNo=0
2012-03-12 16:01:16 [ndbd] INFO     -- Start initiated (mysql-5.5.19 ndb-7.2.4)
NDBFS/AsyncFile: Allocating 310256 for In/Deflate buffer
2012-03-12 16:01:19 [ndbd] INFO     -- timerHandlingLab now: 1360952 sent: 1360682 diff: 270
2012-03-12 16:01:20 [ndbd] WARNING  -- Ndb kernel thread 0 is stuck in: Job Handling elapsed=100
2012-03-12 16:01:20 [ndbd] WARNING  -- Time moved forward with 1813 ms
2012-03-12 16:01:20 [ndbd] INFO     -- Watchdog: User time: 7  System time: 377
2012-03-12 16:01:20 [ndbd] WARNING  -- timerHandlingLab now: 1363000 sent: 1360952 diff: 2048
2012-03-12 16:01:20 [ndbd] WARNING  -- Time moved forward with 1521 ms
2012-03-12 16:01:20 [ndbd] INFO     -- Watchdog: User time: 7  System time: 379
2012-03-12 16:01:20 [ndbd] WARNING  -- Watchdog: Warning overslept 1980 ms, expected 100 ms.
2012-03-12 16:01:21 [ndbd] INFO     -- timerHandlingLab now: 1363825 sent: 1363504 diff: 321
2012-03-12 16:01:21 [ndbd] INFO     -- timerHandlingLab now: 1364008 sent: 1363825 diff: 183
2012-03-12 16:01:21 [ndbd] INFO     -- Watchdog: User time: 7  System time: 404
2012-03-12 16:01:22 [ndbd] WARNING  -- Watchdog: Warning overslept 317 ms, expected 100 ms.
2012-03-12 16:01:22 [ndbd] INFO     -- timerHandlingLab now: 1364384 sent: 1364108 diff: 276
2012-03-12 16:01:22 [ndbd] INFO     -- Watchdog: User time: 7  System time: 418
2012-03-12 16:01:22 [ndbd] WARNING  -- Watchdog: Warning overslept 518 ms, expected 100 ms.
2012-03-12 16:01:22 [ndbd] INFO     -- timerHandlingLab now: 1364806 sent: 1364613 diff: 193
2012-03-12 16:01:25 [ndbd] INFO     -- Watchdog: User time: 7  System time: 450
2012-03-12 16:01:25 [ndbd] WARNING  -- Watchdog: Warning overslept 225 ms, expected 100 ms.
2012-03-12 16:01:25 [ndbd] INFO     -- timerHandlingLab now: 1367439 sent: 1367122 diff: 317
2012-03-12 16:01:25 [ndbd] INFO     -- Watchdog: User time: 7  System time: 451
2012-03-12 16:01:26 [ndbd] ALERT    -- Node 3: Forced node shutdown completed. Occured during startphase 0. Initiated by signal 9.

 

[Tyrant]

I had this same problem a few weeks ago on CentOS 6. It turned out that the sysadmin had left iptables turned on, on one of the data nodes. We just turned off iptables, have you found a solution to your problem yet?

 

[tdktank59]

Yeah the management node did not have enough ram oddly enough...
Upgraded to the next package via linode and it worked fine. (didnt reload the OS or change any settings)

 

[mikeblas]

Less than a gigabyte of memory on a database server? Seriously?

 

[tdktank59]

Working fine now that the management works...

 

[Tawnos]

Less than a gigabyte of memory on a database server? Seriously?

Ram is expensive!

When compared to Ramen

 

[mikeblas]

It's cheaper than disk. And it's lots cheaper than time.

 

[Tawnos]

It's cheaper than disk. And it's lots cheaper than time.

Look for something in dark slate gray in my post . "Ram is expensive" was sarcasm, because you can get 16 gigs of RAM for less than the cost of a nice meal out. Something I'm sure you're acutely aware of.

 

[mikeblas]

And I was agreeing with you.