cluster xl issue

[vinh.nguyen]

HI,

I have an issue with my cluster since i haved test it ...

Since the HA test, i can see that i lost 2 ping every 52 ping on my primary member
When i force the secondary it's worst


i have reboot both of them but still have the ping issue
Can someone help me ?

[ShadowPeak.com]

HI,

I have an issue with my cluster since i haved test it ...

Since the HA test, i can see that i lost 2 ping every 52 ping on my primary member
When i force the secondary it's worst


i have reboot both of them but still have the ping issue
Can someone help me ?

Provide output of "netstat -ni" run on both cluster members after seeing loss please. If you are pinging through networks you do not control (i.e. the Internet) you may not be able to do anything about it.

[vinh.nguyen]

Provide output of "netstat -ni" run on both cluster members after seeing loss please. If you are pinging through networks you do not control (i.e. the Internet) you may not be able to do anything about it.

Hi,


Master
Kernel Interface table
Iface MTU Met RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flg
Exp1-1 1500 0 368009051 0 0 0 967508168 0 0 0 BMRU
Exp1-2 1500 0 1067778749 0 0 0 805837437 0 0 0 BMRU
Exp1-3 1500 0 878850247 0 0 0 668347961 0 0 0 BMRU
Exp1-4 1500 0 331776162 0 0 0 126766382 0 0 0 BMRU
Lan1 1500 0 1157814 0 0 0 1169122 0 0 0 BMRU
Lan2 1500 0 2084523 0 0 0 2042544 0 0 0 BMRU
Lan3 1500 0 272396019 0 225 0 199492823 0 0 0 BMRU
Lan4 1500 0 1238130 0 0 0 1171070 0 0 0 BMRU
Lan5 1500 0 847927773 131217 0 0 973763273 0 0 0 BMRU
Lan6 1500 0 32798662 2 0 0 30683249 0 0 0 BMRU
Lan7 1500 0 0 0 0 0 0 0 0 0 BMU
Lan8 1500 0 1618221 0 0 0 1507739 0 0 0 BMRU
Lan8.11 1500 0 1283893 0 0 0 1227063 0 0 0 BMRU
Mgmt 1500 0 103678192 0 0 0 120850373 0 0 0 BMRU
Sync 1500 0 5725455 0 0 0 22550548 0 0 0 BMRU
lo 16436 0 42263 0 0 0 42263 0 0 0 LRU

Primary
Kernel Interface table
Iface MTU Met RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flg
Exp1-1 1500 0 3573620 0 0 0 5747444 0 0 0 BMRU
Exp1-2 1500 0 15409832 0 865 0 13515402 0 0 0 BMRU
Exp1-3 1500 0 12910144 12480 0 0 8754217 0 0 0 BMRU
Exp1-4 1500 0 1266546 0 0 0 1178003 0 0 0 BMRU
Lan1 1500 0 1151954 0 0 0 1150811 0 0 0 BMRU
Lan2 1500 0 1202923 0 0 0 1151012 0 0 0 BMRU
Lan3 1500 0 3987962 0 0 0 1931070 0 0 0 BMRU
Lan4 1500 0 1217382 0 0 0 1141009 0 0 0 BMRU
Lan5 1500 0 6447338 0 0 0 6317479 0 0 0 BMRU
Lan6 1500 0 1543398 0 0 0 1578552 0 0 0 BMRU
Lan7 1500 0 0 0 0 0 0 0 0 0 BMU
Lan8 1500 0 1172619 0 0 0 1142082 0 0 0 BMRU
Lan8.11 1500 0 1161548 0 0 0 1140160 0 0 0 BMRU
Mgmt 1500 0 6799365 0 0 0 3348529 0 0 0 BMRU
Sync 1500 0 22387323 0 0 0 5649351 0 0 0 BMRU
lo 16436 0 43626 0 0 0 43626 0 0 0 LRU



This is a really strange issue, i lose ping only on the management Interface
sometime i can ping the secondary and and not the master from the lan

[ShadowPeak.com]

Hi,


Master
Kernel Interface table
Iface MTU Met RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flg
Lan5 1500 0 847927773 131217 0 0 973763273 0 0 0 BMRU

Primary
Kernel Interface table
Iface MTU Met RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flg
Exp1-3 1500 0 12910144 12480 0 0 8754217 0 0 0 BMRU

You are racking up network errors on the interfaces above, run "ethtool -S Lan5" on the Master and "ethtool -S Exp1-3" on the Primary and post the results.

[vinh.nguyen]

You are racking up network errors on the interfaces above, run "ethtool -S Lan5" on the Master and "ethtool -S Exp1-3" on the Primary and post the results.

the issue is located on the Mgmt interface

Primary
NIC statistics:
rx_packets: 143547015
tx_packets: 189262442
rx_bytes: 29879897636
tx_bytes: 60504863914
rx_broadcast: 3108893
tx_broadcast: 1335628
rx_multicast: 3032078
tx_multicast: 15
rx_errors: 0
tx_errors: 0
tx_dropped: 0
multicast: 3032078
collisions: 0
rx_length_errors: 0
rx_over_errors: 0
rx_crc_errors: 0
rx_frame_errors: 0
rx_no_buffer_count: 21570
rx_missed_errors: 19706
tx_aborted_errors: 0
tx_carrier_errors: 0
tx_fifo_errors: 0
tx_heartbeat_errors: 0
tx_window_errors: 0
tx_abort_late_coll: 0
tx_deferred_ok: 0
tx_single_coll_ok: 0
tx_multi_coll_ok: 0
tx_timeout_count: 0
tx_restart_queue: 0
rx_long_length_errors: 0
rx_short_length_errors: 0
rx_align_errors: 0
tx_tcp_seg_good: 0
tx_tcp_seg_failed: 0
rx_flow_control_xon: 0
rx_flow_control_xoff: 0
tx_flow_control_xon: 0
tx_flow_control_xoff: 0
rx_long_byte_count: 29879897636
rx_csum_offload_good: 30148414
rx_csum_offload_errors: 0
rx_header_split: 0
alloc_rx_buff_failed: 0
tx_smbus: 0
rx_smbus: 63869
dropped_smbus: 0

Secondary
NIC statistics:
rx_packets: 7729382
tx_packets: 4116624
rx_bytes: 897967112
tx_bytes: 1739957384
rx_broadcast: 3139693
tx_broadcast: 1300138
rx_multicast: 3019714
tx_multicast: 21
rx_errors: 0
tx_errors: 0
tx_dropped: 0
multicast: 3019714
collisions: 0
rx_length_errors: 0
rx_over_errors: 0
rx_crc_errors: 0
rx_frame_errors: 0
rx_no_buffer_count: 0
rx_missed_errors: 0
tx_aborted_errors: 0
tx_carrier_errors: 0
tx_fifo_errors: 0
tx_heartbeat_errors: 0
tx_window_errors: 0
tx_abort_late_coll: 0
tx_deferred_ok: 0
tx_single_coll_ok: 0
tx_multi_coll_ok: 0
tx_timeout_count: 0
tx_restart_queue: 0
rx_long_length_errors: 0
rx_short_length_errors: 0
rx_align_errors: 0
tx_tcp_seg_good: 0
tx_tcp_seg_failed: 0
rx_flow_control_xon: 0
rx_flow_control_xoff: 0
tx_flow_control_xon: 0
tx_flow_control_xoff: 0
rx_long_byte_count: 897967112
rx_csum_offload_good: 6754465
rx_csum_offload_errors: 0
rx_header_split: 0
alloc_rx_buff_failed: 0
tx_smbus: 0
rx_smbus: 73935
dropped_smbus: 0

[ShadowPeak.com]

The management interface issues shouldn't cause production packet loss unless your ping traffic is crossing that interface. Try a ping test from a system directly attached to the same VLAN as a firewall interface, to a system directly attached to the same VLAN as the firewall's interface on the target side. Also check for switch port counters for errors on the following ports:

ping initiating system
firewall ingress interface
firewall egress interface
target system

It is far more likely you have some kind of issue with your switch infrastructure, especially considering you are clearly having some of those issues on the management interface. Once all this has been vetted we can focus on the firewall itself.

[vinh.nguyen]

The management interface issues shouldn't cause production packet loss unless your ping traffic is crossing that interface. Try a ping test from a system directly attached to the same VLAN as a firewall interface, to a system directly attached to the same VLAN as the firewall's interface on the target side. Also check for switch port counters for errors on the following ports:

ping initiating system
firewall ingress interface
firewall egress interface
target system

It is far more likely you have some kind of issue with your switch infrastructure, especially considering you are clearly having some of those issues on the management interface. Once all this has been vetted we can focus on the firewall itself.

Hi,
Thanks for the advice, i have already check the switch

i was Receiving to many broadcast UDP packets on port 8116

[ShadowPeak.com]

Hi,
Thanks for the advice, i have already check the switch

i was Receiving to many broadcast UDP packets on port 8116

UDP 8116 is Cluster Control Protocol and it is normal to see lots of those.

Next step is to run a tcpdump so see if the firewall is actually eating the traffic, assume that during your ping test 1.2.3.4 is pinging 129.82.102.32, run this command from expert mode on the firewall and launch your ping test:

tcpdump -eni any icmp and host 1.2.3.4 and host 129.82.102.32

• Seeing echo requests arrive but not leave in the tcpdump output indicates the firewall is eating it, next step is running "fw ctl zdebug drop" during the ping test to see why.
• Missing echo requests (should see one roughly every second) means issue is on the ingress side: check for switchport errors, spanning-tree flaps or traffic policing
• Seeing echo requests arrive, leave, and no echo reply coming back indicates an issue on the egress side: check for switchport errors, spanning-tree flaps or traffic policing
• Seeing echo requests arrive/leave, echo reply comes back but doesn't leave indicates the firewall is eating it, use fw ctl zdebug drop command as described above.

Also keep an eye on the MAC addresses displayed by tcpdump and ensure they match the firewall, they should remain constant by direction and not shift around which could indicate a IP conflict with the firewall.

[cciesec2006]

The management interface issues shouldn't cause production packet loss unless your ping traffic is crossing that interface. Try a ping test from a system directly attached to the same VLAN as a firewall interface, to a system directly attached to the same VLAN as the firewall's interface on the target side. Also check for switch port counters for errors on the following ports:

ping initiating system
firewall ingress interface
firewall egress interface
target system

It is far more likely you have some kind of issue with your switch infrastructure, especially considering you are clearly having some of those issues on the management interface. Once all this has been vetted we can focus on the firewall itself.

Have you ever thought that the issue could be related to "microburst"? I don't know if it happens in your environment but it is possible:

"Identifying microbursts or any bursty traffic is a good example of why it’s important to ‘know what you don’t know’. If someone complains about seeing issues on a link it’s important not to immediately dismiss the complaint and do some due diligence. While monitoring interface statistics via SNMP in 1 or 5 minute intervals is an excellent start, it’s important to know that there may be things going on in the network that aren’t showing up in those graphs. By utilizing a number of different tools you can trace down problems. Reducing the interface load-interval to 30 seconds and tracking your output drops is a good start.Using Wireshark allows you to dive further into the problem and figure out what traffic is causing or contributing to the drops."