CPUG: The Check Point User Group

Resources for the Check Point Community, by the Check Point Community.


First, I hope you're all well and staying safe.
Second, I want to give a "heads up" that you should see more activity here shortly, and maybe a few cosmetic changes.
I'll post more details to the "Announcements" forum soon, so be on the lookout. -E

 

Results 1 to 9 of 9

Thread: cluster xl issue

  1. #1
    Join Date
    2016-10-28
    Location
    Paris
    Posts
    19
    Rep Power
    0

    Default cluster xl issue

    HI,

    I have an issue with my cluster since i haved test it ...

    Since the HA test, i can see that i lost 2 ping every 52 ping on my primary member
    When i force the secondary it's worst


    i have reboot both of them but still have the ping issue
    Can someone help me ?

  2. #2
    Join Date
    2009-04-30
    Location
    Colorado, USA
    Posts
    2,252
    Rep Power
    17

    Default Re: cluster xl issue

    Quote Originally Posted by vinh.nguyen View Post
    HI,

    I have an issue with my cluster since i haved test it ...

    Since the HA test, i can see that i lost 2 ping every 52 ping on my primary member
    When i force the secondary it's worst


    i have reboot both of them but still have the ping issue
    Can someone help me ?
    Provide output of "netstat -ni" run on both cluster members after seeing loss please. If you are pinging through networks you do not control (i.e. the Internet) you may not be able to do anything about it.
    --
    Third Edition of my "Max Power 2020" Firewall Book
    Now Available at http://www.maxpowerfirewalls.com

  3. #3
    Join Date
    2016-10-28
    Location
    Paris
    Posts
    19
    Rep Power
    0

    Default Re: cluster xl issue

    Quote Originally Posted by ShadowPeak.com View Post
    Provide output of "netstat -ni" run on both cluster members after seeing loss please. If you are pinging through networks you do not control (i.e. the Internet) you may not be able to do anything about it.

    Hi,


    Master
    Kernel Interface table
    Iface MTU Met RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flg
    Exp1-1 1500 0 368009051 0 0 0 967508168 0 0 0 BMRU
    Exp1-2 1500 0 1067778749 0 0 0 805837437 0 0 0 BMRU
    Exp1-3 1500 0 878850247 0 0 0 668347961 0 0 0 BMRU
    Exp1-4 1500 0 331776162 0 0 0 126766382 0 0 0 BMRU
    Lan1 1500 0 1157814 0 0 0 1169122 0 0 0 BMRU
    Lan2 1500 0 2084523 0 0 0 2042544 0 0 0 BMRU
    Lan3 1500 0 272396019 0 225 0 199492823 0 0 0 BMRU
    Lan4 1500 0 1238130 0 0 0 1171070 0 0 0 BMRU
    Lan5 1500 0 847927773 131217 0 0 973763273 0 0 0 BMRU
    Lan6 1500 0 32798662 2 0 0 30683249 0 0 0 BMRU
    Lan7 1500 0 0 0 0 0 0 0 0 0 BMU
    Lan8 1500 0 1618221 0 0 0 1507739 0 0 0 BMRU
    Lan8.11 1500 0 1283893 0 0 0 1227063 0 0 0 BMRU
    Mgmt 1500 0 103678192 0 0 0 120850373 0 0 0 BMRU
    Sync 1500 0 5725455 0 0 0 22550548 0 0 0 BMRU
    lo 16436 0 42263 0 0 0 42263 0 0 0 LRU

    Primary
    Kernel Interface table
    Iface MTU Met RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flg
    Exp1-1 1500 0 3573620 0 0 0 5747444 0 0 0 BMRU
    Exp1-2 1500 0 15409832 0 865 0 13515402 0 0 0 BMRU
    Exp1-3 1500 0 12910144 12480 0 0 8754217 0 0 0 BMRU
    Exp1-4 1500 0 1266546 0 0 0 1178003 0 0 0 BMRU
    Lan1 1500 0 1151954 0 0 0 1150811 0 0 0 BMRU
    Lan2 1500 0 1202923 0 0 0 1151012 0 0 0 BMRU
    Lan3 1500 0 3987962 0 0 0 1931070 0 0 0 BMRU
    Lan4 1500 0 1217382 0 0 0 1141009 0 0 0 BMRU
    Lan5 1500 0 6447338 0 0 0 6317479 0 0 0 BMRU
    Lan6 1500 0 1543398 0 0 0 1578552 0 0 0 BMRU
    Lan7 1500 0 0 0 0 0 0 0 0 0 BMU
    Lan8 1500 0 1172619 0 0 0 1142082 0 0 0 BMRU
    Lan8.11 1500 0 1161548 0 0 0 1140160 0 0 0 BMRU
    Mgmt 1500 0 6799365 0 0 0 3348529 0 0 0 BMRU
    Sync 1500 0 22387323 0 0 0 5649351 0 0 0 BMRU
    lo 16436 0 43626 0 0 0 43626 0 0 0 LRU



    This is a really strange issue, i lose ping only on the management Interface
    sometime i can ping the secondary and and not the master from the lan

  4. #4
    Join Date
    2009-04-30
    Location
    Colorado, USA
    Posts
    2,252
    Rep Power
    17

    Default Re: cluster xl issue

    Quote Originally Posted by vinh.nguyen View Post
    Hi,


    Master
    Kernel Interface table
    Iface MTU Met RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flg
    Lan5 1500 0 847927773 131217 0 0 973763273 0 0 0 BMRU

    Primary
    Kernel Interface table
    Iface MTU Met RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flg
    Exp1-3 1500 0 12910144 12480 0 0 8754217 0 0 0 BMRU
    You are racking up network errors on the interfaces above, run "ethtool -S Lan5" on the Master and "ethtool -S Exp1-3" on the Primary and post the results.
    --
    Third Edition of my "Max Power 2020" Firewall Book
    Now Available at http://www.maxpowerfirewalls.com

  5. #5
    Join Date
    2016-10-28
    Location
    Paris
    Posts
    19
    Rep Power
    0

    Default Re: cluster xl issue

    Quote Originally Posted by ShadowPeak.com View Post
    You are racking up network errors on the interfaces above, run "ethtool -S Lan5" on the Master and "ethtool -S Exp1-3" on the Primary and post the results.
    the issue is located on the Mgmt interface

    Primary
    NIC statistics:
    rx_packets: 143547015
    tx_packets: 189262442
    rx_bytes: 29879897636
    tx_bytes: 60504863914
    rx_broadcast: 3108893
    tx_broadcast: 1335628
    rx_multicast: 3032078
    tx_multicast: 15
    rx_errors: 0
    tx_errors: 0
    tx_dropped: 0
    multicast: 3032078
    collisions: 0
    rx_length_errors: 0
    rx_over_errors: 0
    rx_crc_errors: 0
    rx_frame_errors: 0
    rx_no_buffer_count: 21570
    rx_missed_errors: 19706
    tx_aborted_errors: 0
    tx_carrier_errors: 0
    tx_fifo_errors: 0
    tx_heartbeat_errors: 0
    tx_window_errors: 0
    tx_abort_late_coll: 0
    tx_deferred_ok: 0
    tx_single_coll_ok: 0
    tx_multi_coll_ok: 0
    tx_timeout_count: 0
    tx_restart_queue: 0
    rx_long_length_errors: 0
    rx_short_length_errors: 0
    rx_align_errors: 0
    tx_tcp_seg_good: 0
    tx_tcp_seg_failed: 0
    rx_flow_control_xon: 0
    rx_flow_control_xoff: 0
    tx_flow_control_xon: 0
    tx_flow_control_xoff: 0
    rx_long_byte_count: 29879897636
    rx_csum_offload_good: 30148414
    rx_csum_offload_errors: 0
    rx_header_split: 0
    alloc_rx_buff_failed: 0
    tx_smbus: 0
    rx_smbus: 63869
    dropped_smbus: 0

    Secondary
    NIC statistics:
    rx_packets: 7729382
    tx_packets: 4116624
    rx_bytes: 897967112
    tx_bytes: 1739957384
    rx_broadcast: 3139693
    tx_broadcast: 1300138
    rx_multicast: 3019714
    tx_multicast: 21
    rx_errors: 0
    tx_errors: 0
    tx_dropped: 0
    multicast: 3019714
    collisions: 0
    rx_length_errors: 0
    rx_over_errors: 0
    rx_crc_errors: 0
    rx_frame_errors: 0
    rx_no_buffer_count: 0
    rx_missed_errors: 0
    tx_aborted_errors: 0
    tx_carrier_errors: 0
    tx_fifo_errors: 0
    tx_heartbeat_errors: 0
    tx_window_errors: 0
    tx_abort_late_coll: 0
    tx_deferred_ok: 0
    tx_single_coll_ok: 0
    tx_multi_coll_ok: 0
    tx_timeout_count: 0
    tx_restart_queue: 0
    rx_long_length_errors: 0
    rx_short_length_errors: 0
    rx_align_errors: 0
    tx_tcp_seg_good: 0
    tx_tcp_seg_failed: 0
    rx_flow_control_xon: 0
    rx_flow_control_xoff: 0
    tx_flow_control_xon: 0
    tx_flow_control_xoff: 0
    rx_long_byte_count: 897967112
    rx_csum_offload_good: 6754465
    rx_csum_offload_errors: 0
    rx_header_split: 0
    alloc_rx_buff_failed: 0
    tx_smbus: 0
    rx_smbus: 73935
    dropped_smbus: 0

  6. #6
    Join Date
    2009-04-30
    Location
    Colorado, USA
    Posts
    2,252
    Rep Power
    17

    Default Re: cluster xl issue

    The management interface issues shouldn't cause production packet loss unless your ping traffic is crossing that interface. Try a ping test from a system directly attached to the same VLAN as a firewall interface, to a system directly attached to the same VLAN as the firewall's interface on the target side. Also check for switch port counters for errors on the following ports:

    ping initiating system
    firewall ingress interface
    firewall egress interface
    target system

    It is far more likely you have some kind of issue with your switch infrastructure, especially considering you are clearly having some of those issues on the management interface. Once all this has been vetted we can focus on the firewall itself.
    --
    Third Edition of my "Max Power 2020" Firewall Book
    Now Available at http://www.maxpowerfirewalls.com

  7. #7
    Join Date
    2016-10-28
    Location
    Paris
    Posts
    19
    Rep Power
    0

    Default Re: cluster xl issue

    Quote Originally Posted by ShadowPeak.com View Post
    The management interface issues shouldn't cause production packet loss unless your ping traffic is crossing that interface. Try a ping test from a system directly attached to the same VLAN as a firewall interface, to a system directly attached to the same VLAN as the firewall's interface on the target side. Also check for switch port counters for errors on the following ports:

    ping initiating system
    firewall ingress interface
    firewall egress interface
    target system

    It is far more likely you have some kind of issue with your switch infrastructure, especially considering you are clearly having some of those issues on the management interface. Once all this has been vetted we can focus on the firewall itself.

    Hi,
    Thanks for the advice, i have already check the switch

    i was Receiving to many broadcast UDP packets on port 8116

  8. #8
    Join Date
    2009-04-30
    Location
    Colorado, USA
    Posts
    2,252
    Rep Power
    17

    Default Re: cluster xl issue

    Quote Originally Posted by vinh.nguyen View Post
    Hi,
    Thanks for the advice, i have already check the switch

    i was Receiving to many broadcast UDP packets on port 8116
    UDP 8116 is Cluster Control Protocol and it is normal to see lots of those.

    Next step is to run a tcpdump so see if the firewall is actually eating the traffic, assume that during your ping test 1.2.3.4 is pinging 129.82.102.32, run this command from expert mode on the firewall and launch your ping test:

    tcpdump -eni any icmp and host 1.2.3.4 and host 129.82.102.32

    • Seeing echo requests arrive but not leave in the tcpdump output indicates the firewall is eating it, next step is running "fw ctl zdebug drop" during the ping test to see why.
    • Missing echo requests (should see one roughly every second) means issue is on the ingress side: check for switchport errors, spanning-tree flaps or traffic policing
    • Seeing echo requests arrive, leave, and no echo reply coming back indicates an issue on the egress side: check for switchport errors, spanning-tree flaps or traffic policing
    • Seeing echo requests arrive/leave, echo reply comes back but doesn't leave indicates the firewall is eating it, use fw ctl zdebug drop command as described above.


    Also keep an eye on the MAC addresses displayed by tcpdump and ensure they match the firewall, they should remain constant by direction and not shift around which could indicate a IP conflict with the firewall.
    --
    Third Edition of my "Max Power 2020" Firewall Book
    Now Available at http://www.maxpowerfirewalls.com

  9. #9
    Join Date
    2006-09-26
    Posts
    3,200
    Rep Power
    20

    Default Re: cluster xl issue

    Quote Originally Posted by ShadowPeak.com View Post
    The management interface issues shouldn't cause production packet loss unless your ping traffic is crossing that interface. Try a ping test from a system directly attached to the same VLAN as a firewall interface, to a system directly attached to the same VLAN as the firewall's interface on the target side. Also check for switch port counters for errors on the following ports:

    ping initiating system
    firewall ingress interface
    firewall egress interface
    target system

    It is far more likely you have some kind of issue with your switch infrastructure, especially considering you are clearly having some of those issues on the management interface. Once all this has been vetted we can focus on the firewall itself.
    Have you ever thought that the issue could be related to "microburst"? I don't know if it happens in your environment but it is possible:

    "Identifying microbursts or any bursty traffic is a good example of why itís important to Ďknow what you donít knowí. If someone complains about seeing issues on a link itís important not to immediately dismiss the complaint and do some due diligence. While monitoring interface statistics via SNMP in 1 or 5 minute intervals is an excellent start, itís important to know that there may be things going on in the network that arenít showing up in those graphs. By utilizing a number of different tools you can trace down problems. Reducing the interface load-interval to 30 seconds and tracking your output drops is a good start.Using Wireshark allows you to dive further into the problem and figure out what traffic is causing or contributing to the drops."

Similar Threads

  1. Replies: 2
    Last Post: 2015-12-29, 04:55
  2. Cluster issue
    By nileshwagh20000 in forum Advanced Networking & Clustering Blade
    Replies: 11
    Last Post: 2013-02-12, 07:09
  3. Cluster issue
    By nileshwagh20000 in forum Firewall Blade
    Replies: 3
    Last Post: 2013-02-04, 08:40
  4. Cluster Issue,
    By 013rgk in forum Clustering (Security Gateway HA and ClusterXL)
    Replies: 4
    Last Post: 2012-05-04, 05:40
  5. Cluster issue in UTM
    By vijay_vya in forum Clustering (Security Gateway HA and ClusterXL)
    Replies: 1
    Last Post: 2010-05-03, 18:26

Bookmarks

Posting Permissions

  • You may not post new threads
  • You may not post replies
  • You may not post attachments
  • You may not edit your posts
  •