[7.250] Webadmin shows "ha_system_unstable" [CONFIRMED]

[BuBU]

Hi,

I've just installed a new asg box and activated HA Active/passive on the first asg box... then activated HA cluster active/passive on the second box using eth0 for interco between both firewalls...
I received this email:

HA System is active and working

-- 
HA Status          : HA MASTER (node id: 1)
System Uptime      : 0 days 9 hours 51 minutes
System Load        : 0.10
System Version     : Astaro Security Gateway Software 7.250

Please refer to the manual for detailed instructions.

and I see this into the HA Status:

1 	MASTER 	asg1 	ACTIVE 	7.250 	Thu Jul 10 22:40:03 2008  
2 	SLAVE 	asg2 	ACTIVE 	7.250 	Thu Jul 10 22:53:42 2008

then trying to shutdown the MASTER results into an error message:

HA_SYSTEM_UNSTABLE,

and the master don't want to shutdown..

will try to shutdown it through the console..

[gkemter]

i think, u left no time for full sync between master/slave.
After add a slave , the cluster needs ~ 20min to sync.


Gregor Kemter

[BuBU]

I've shutdowned asg1 (the master)... and the slave asg2 did not come up... when I connected on the console I still see old ip for eth2 I defined when installed... but got <M> so it goes master... but seems like config got not replicated

eth0 (which is interco) get ip 198.19.250.2 on asg2 and 198.19.250.1 on asg1 and each other are pingable...

so there is something wrong...

[BuBU]

Originally Posted by gkemter

i think, u left no time for full sync between master/slave.
After add a slave , the cluster needs ~ 20min to sync.


Gregor Kemter

probably but in that case it must not say ACTIVE/READY/WORKING !!!

[gkemter]

prior 7.250 u got a messagebox : Not Sync or something, if u try to reboot from webmin before sync was ready.

[da_merlin]

Originally Posted by BuBU

I've shutdowned asg1 (the master)... and the slave asg2 did not come up... when I connected on the console I still see old ip for eth2 I defined when installed... but got <M> so it goes master... but seems like config got not replicated

eth0 (which is interco) get ip 198.19.250.2 on asg2 and 198.19.250.1 on asg1 and each other are pingable...

so there is something wrong...

Is there anything in /var/log/confd.log from confd-sync daemon?

[BuBU]

I've just started reinstall everything

at moment I've only master installed and I get this in HA log:

2008:07:11-12:54:59 (none) slon[11509]: [2-1] ERROR cannot get sl_local_node_id - ERROR: schema "_asg_cluster" does not exist
2008:07:11-12:54:59 (none) slon[11509]: [3-1] FATAL main: Node is not initialized properly - sleep 10s
2008:07:11-12:54:59 (none) slon[11511]: [2-1] ERROR cannot get sl_local_node_id - ERROR: schema "_asg_cluster" does not exist
2008:07:11-12:54:59 (none) slon[11511]: [3-1] FATAL main: Node is not initialized properly - sleep 10s

is that normal ? seems slony did not initialize itself correctly ?

[da_merlin]

As long as there is no Slave around, Slony will log these messages. This is normal.

Start the Slave system and ensure the configuration is synced properly.
(e.g. IP addresses of all interfaces, hostname).
If this fails, watch for confd-sync messages and post them here...

[BuBU]

ok so I've reinstalled asg1 and activated HA active/passive... then installed asg2 and activated HA active/passive

so at this stage asg1 is MASTER and asg2 SLAVE..

when checking interfaces on asg2/SLAVE it shows same iface as on asg1 (except ip for the interco)

so I did shutdown asg1/MASTER so after about 20secs asg2 became new MASTER which is fine... and internet is working again so this is good..

then I started asg1 which became SLAVE as expected... and after a while I tried to shutdown asg2/MASTER and got the ha_system_unstable

here are logs for confd-sync (grep from /var/log/confd.log):

for asg1:
http://www.gmlinux.org/confd-sync-asg1.log

for asg2:
http://www.gmlinux.org/confd-sync-asg2.log

I think I've waited enough time... will check later... else this can be critical when in real production !

[da_merlin]

Originally Posted by BuBU

I think I've waited enough time... will check later... else this can be critical when in real production !

There is a 15 minute time lock after a Slave system booted up, before the Master system can be rebooted via WebAdmin.

This issue is NOT critical in real production because if you power off your Master system within these 15 minutes, there will be a Takeover!