PDA

View Full Version : Checkpoint CPU question



oharek
2017-12-14, 09:46
Hello,

I have got 2 x Checkpoint 4400's externally facing with 100MB links to the internet. Recently we are using more applications like Office365 and Skype. Skype calls drop out occasionally

The CPU on each device can fluctuate from 30 to 90% very quickly.

I think i have 3 options but dont know which would solve the puzzle - i was hoping someone else might have had a similar issue

Is this because of IPS signatures i have running now also even though i have tried to keep them to a minimum.
Maybe i need to upgrade the Checkpoint 4400 to a better spec
Thirdly - maybe i just need to ramp up the internet pipe to 200 or 300MB



thanks

ShadowPeak.com
2017-12-14, 11:59
Hello,

I have got 2 x Checkpoint 4400's externally facing with 100MB links to the internet. Recently we are using more applications like Office365 and Skype. Skype calls drop out occasionally

The CPU on each device can fluctuate from 30 to 90% very quickly.

I think i have 3 options but dont know which would solve the puzzle - i was hoping someone else might have had a similar issue

Is this because of IPS signatures i have running now also even though i have tried to keep them to a minimum.
Maybe i need to upgrade the Checkpoint 4400 to a better spec
Thirdly - maybe i just need to ramp up the internet pipe to 200 or 300MB



thanks

Firewall code & HFA version?

Also if you provide the output of all these commands I should be able to provide some advice:

fwaccel stat
fwaccel stats -s
fw ctl affinity -l -r
sim affinity -l
netstat -ni
fw ctl multik stat
cpstat os -f multi_cpu -o 1
free -m
fw ctl multik get_mode
fw ctl pstat

--
My Book "Max Power: Check Point Firewall Performance Optimization"
Second Edition Coming Soon

oharek
2017-12-15, 06:15
One of the checkpoints is sitting at 95% today and the other is 555
I have attached the output from the one of the two checkpoints which is 95%

Any advice is welcome - i am just trying to work out do i need to upgrade the checkpoint hardware or the link itself

Firewall code


HFA version - Version, R77.30



Also if you provide the output of all these commands I should be able to provide some advice:

[Expert@UTM-KOH-CORP:0]# fwaccel stat
Accelerator Status : on
Accept Templates : enabled
Drop Templates : disabled
NAT Templates : disabled by user

Accelerator Features : Accounting, NAT, Cryptography, Routing,
HasClock, Templates, Synchronous, IdleDetection,
Sequencing, TcpStateDetect, AutoExpire,
DelayedNotif, TcpStateDetectV2, CPLS, McastRouting,
WireMode, DropTemplates, NatTemplates,
Streaming, MultiFW, AntiSpoofing, Nac,
ViolationStats, AsychronicNotif, ERDOS,
NAT64, GTPAcceleration, SCTPAcceleration,
McastRoutingV2
Cryptography Features : Tunnel, UDPEncapsulation, MD5, SHA1, NULL,
3DES, DES, CAST, CAST-40, AES-128, AES-256,
ESP, LinkSelection, DynamicVPN, NatTraversal,
EncRouting, AES-XCBC, SHA256

[Expert@UTM-KOH-CORP:0]# fwaccel stats -s
Accelerated conns/Total conns : 0/3316 (0%)
Accelerated pkts/Total pkts : 3014/111229358 (0%)
F2Fed pkts/Total pkts : 104999123/111229358 (94%)
PXL pkts/Total pkts : 6227221/111229358 (5%)
QXL pkts/Total pkts : 0/111229358 (0%)

[Expert@UTM-KOH-CORP:0]# fw ctl affinity -l -r
CPU 0: eth1 eth2 eth3 Mgmt
fw_1
CPU 1: fw_0
All: rtmd usrchkd in.geod fwd rad mpdaemon vpnd cpd cprid

[Expert@UTM-KOH-CORP:0]# sim affinity -l
Mgmt : 0
eth1 : 0
eth2 : 0
eth3 : 0


[Expert@UTM-KOH-CORP:0]# netstat -ni
Kernel Interface table
Iface MTU Met RX-OK RX-ERR RX-DRP RX-OVR TX-OK TX-ERR TX-DRP TX-OVR Flg
Mgmt 1500 0 13618688 1 0 0 18398436 0 0 0 BMRU
eth1 1500 0 2867376571 0 305369 0 2609766624 0 0 0 BMRU
eth2 1500 0 2568837649 0 469382 0 2783466204 0 0 0 BMRU
eth3 1500 0 3680048 0 0 0 56009 0 0 0 BMRU
eth3.100 1500 0 3679634 0 0 0 55621 0 0 0 BMRU
eth3.130 1500 0 157 0 0 0 132 0 0 0 BMRU
eth3.131 1500 0 257 0 0 0 256 0 0 0 BMRU
eth3.193 1500 0 0 0 0 0 0 0 0 0 BMRU
eth3.194 1500 0 0 0 0 0 0 0 0 0 BMRU
lo 16436 0 44267478 0 0 0 44267478 0 0 0 LRU



Processors load
---------------------------------------------------------------------------------
|CPU#|User Time(%)|System Time(%)|Idle Time(%)|Usage(%)|Run queue|Interrupts/sec|
---------------------------------------------------------------------------------
| 1| 6| 95| 0| 100| ?| 3247|
| 2| 2| 87| 12| 88| ?| 3247|
---------------------------------------------------------------------------------





Processors load
---------------------------------------------------------------------------------
|CPU#|User Time(%)|System Time(%)|Idle Time(%)|Usage(%)|Run queue|Interrupts/sec|
---------------------------------------------------------------------------------
| 1| 2| 98| 0| 100| ?| 1854|
| 2| 3| 86| 12| 88| ?| 1854|
---------------------------------------------------------------------------------





Processors load
---------------------------------------------------------------------------------
|CPU#|User Time(%)|System Time(%)|Idle Time(%)|Usage(%)|Run queue|Interrupts/sec|
---------------------------------------------------------------------------------
| 1| 2| 98| 0| 100| ?| 1854|
| 2| 3| 86| 12| 88| ?| 1854|
---------------------------------------------------------------------------------





Processors load
---------------------------------------------------------------------------------
|CPU#|User Time(%)|System Time(%)|Idle Time(%)|Usage(%)|Run queue|Interrupts/sec|
---------------------------------------------------------------------------------
| 1| 11| 89| 1| 99| ?| 6922|
| 2| 2| 85| 12| 88| ?| 3461|
---------------------------------------------------------------------------------
Processors load
---------------------------------------------------------------------------------
|CPU#|User Time(%)|System Time(%)|Idle Time(%)|Usage(%)|Run queue|Interrupts/sec|
---------------------------------------------------------------------------------
| 1| 11| 89| 1| 99| ?| 6922|
| 2| 2| 85| 12| 88| ?| 3461|
---------------------------------------------------------------------------------





Processors load
---------------------------------------------------------------------------------
|CPU#|User Time(%)|System Time(%)|Idle Time(%)|Usage(%)|Run queue|Interrupts/sec|
---------------------------------------------------------------------------------
| 1| 1| 99| 0| 100| ?| 2225|
| 2| 11| 84| 6| 94| ?| 2225|
---------------------------------------------------------------------------------





Processors load
---------------------------------------------------------------------------------
|CPU#|User Time(%)|System Time(%)|Idle Time(%)|Usage(%)|Run queue|Interrupts/sec|
---------------------------------------------------------------------------------
| 1| 1| 99| 0| 100| ?| 2225|
| 2| 11| 84| 6| 94| ?| 2225|
---------------------------------------------------------------------------------





Processors load
---------------------------------------------------------------------------------
|CPU#|User Time(%)|System Time(%)|Idle Time(%)|Usage(%)|Run queue|Interrupts/sec|
---------------------------------------------------------------------------------
| 1| 4| 95| 2| 98| ?| 3593|
| 2| 2| 87| 10| 90| ?| 3593|
---------------------------------------------------------------------------------

[Expert@UTM-KOH-CORP:0]# free -m
total used free shared buffers cached
Mem: 3948 3122 826 0 247 1194
-/+ buffers/cache: 1680 2267
Swap: 10268 0 10267



[Expert@UTM-KOH-CORP:0]# fw ctl multik get_mode
Current mode is Off


[Expert@UTM-KOH-CORP:0]# fw ctl pstat

System Capacity Summary:
Memory used: 25% (405 MB out of 1587 MB) - below watermark
Concurrent Connections: 3474 (Unlimited)
Aggressive Aging is not active

Hash kernel memory (hmem) statistics:
Total memory allocated: 343932928 bytes in 83968 (4096 bytes) blocks using 82 pools
Initial memory allocated: 163577856 bytes (Hash memory extended by 180355072 bytes)
Memory allocation limit: 831520768 bytes using 512 pools
Total memory bytes used: 149164196 unused: 194768732 (56.63%) peak: 323774744
Total memory blocks used: 42910 unused: 41058 (48%) peak: 79978
Allocations: 1817907171 alloc, 0 failed alloc, 1816286913 free

System kernel memory (smem) statistics:
Total memory bytes used: 493127164 peak: 566746496
Total memory bytes wasted: 31263512
Blocking memory bytes used: 8879524 peak: 26352252
Non-Blocking memory bytes used: 484247640 peak: 540394244
Allocations: 23690120 alloc, 0 failed alloc, 23686365 free, 0 failed free
vmalloc bytes used: 18730292 expensive: yes

Kernel memory (kmem) statistics:
Total memory bytes used: 297360796 peak: 511186200
Allocations: 1841590458 alloc, 0 failed alloc
1839967336 free, 0 failed free
External Allocations: 225824 for packets, 48207501 for SXL

Cookies:
4152387766 total, 1799 alloc, 1799 free,
1882195 dup, 2305949662 get, 1909900152 put,
4105096625 len, 663 cached len, 0 chain alloc,
0 chain free

Connections:
29814286 total, 16091836 TCP, 12842866 UDP, 879581 ICMP,
3 other, 98659 anticipated, 30963 recovered, 3474 concurrent,
13621 peak concurrent

Fragments:
8 fragments, 4 packets, 0 expired, 0 short,
0 large, 0 duplicates, 0 failures

NAT:
-1942571066/0 forw, 1865165534/0 bckw, 1863895345 tcpudp,
1729082 icmp, 29614294-29762081 alloc

ShadowPeak.com
2017-12-15, 09:14
[Expert@UTM-KOH-CORP:0]# fwaccel stats -s
Accelerated conns/Total conns : 0/3316 (0%)
Accelerated pkts/Total pkts : 3014/111229358 (0%)
F2Fed pkts/Total pkts : 104999123/111229358 (94%)
PXL pkts/Total pkts : 6227221/111229358 (5%)
QXL pkts/Total pkts : 0/111229358 (0%)



1) Right there is your main issue. SecureXL is on but you are getting practically zero acceleration or templating. Maybe something we can fix and improve performance quite a bit, please provide output of:

enabled_blades
installed_jumbo_take
cpinfo -y

It is *probably* your IPS configuration causing this, but we will see.

2) Dynamic Dispatcher is off, might help a little bit to enable it but we should resolve the non-acceleration issue first.

3) You said you had two firewalls are they in a cluster? Doesn't look like these commands were run on a cluster member as there are no state sync stats.

4) Are you using the ISP Redundancy feature? HTTPS Inspection?

RX-DRP packet loss is within acceptable limits.

oharek
2017-12-15, 10:50
[Expert@UTM-WEST-CORP:0]# enabled_blades
fw appi ips



[Expert@UTM-WEST-CORP:0]# installed_jumbo_take
bash: installed_jumbo_take: command not found




[Expert@UTM-WEST-CORP:0]# cpinfo -y

------------------------
Hotfix versions
------------------------
[FW1]
HOTFIX_R77_30

[SecurePlatform]
No hotfixes..

[CPinfo]
No hotfixes..

[PPACK]
HOTFIX_R77_30

[CVPN]
HOTFIX_R77_30

[rtm]
No hotfixes..


I have two Checkpoint Firewalls running as standalone boxes at two different sites
but they are both on the same infrastructure so i use a checkpoint manager 3050 to push the same poicy and IPS to both so no clustering is setup or required

i am not using IPS redundancy or HTTPS inspection

ShadowPeak.com
2017-12-15, 11:04
[Expert@UTM-WEST-CORP:0]# enabled_blades
fw appi ips



[Expert@UTM-WEST-CORP:0]# installed_jumbo_take
bash: installed_jumbo_take: command not found




[Expert@UTM-WEST-CORP:0]# cpinfo -y

------------------------
Hotfix versions
------------------------
[FW1]
HOTFIX_R77_30

[SecurePlatform]
No hotfixes..

[CPinfo]
No hotfixes..

[PPACK]
HOTFIX_R77_30

[CVPN]
HOTFIX_R77_30

[rtm]
No hotfixes..


I have two Checkpoint Firewalls running as standalone boxes at two different sites
but they are both on the same infrastructure so i use a checkpoint manager 3050 to push the same poicy and IPS to both so no clustering is setup or required

i am not using IPS redundancy or HTTPS inspection

1) Hmm really should install the latest GA Jumbo HFA for R77.30 (Take 286) but not really required to help solve your performance problem given the limited number of blades you have enabled.

2) So it is almost certainly IPS causing the high F2F, what IPS profile are you using? Try switching over to the Default_Protection IPS profile; if you haven't changed anything in that profile, all traffic inspected by IPS with this profile active is eligible for acceleration. Once you've installed policy with the new IPS profile set, run fwaccel stats -r, wait 10 minutes, then run fwaccel stats -s again, check the new CPU load as well, and post the results.

3) Check your APCL policy, you should NOT be using Any in the destination of any APCL rule; you don't need the final "Any Any Any Recognized Accept" in the APCL policy either (unless you want highly detailed logging) because the default action if no APCL rule is matched is Accept. You should only be using object Internet in the destination of all APCL rules, otherwise LAN-speed traffic between internal networks and/or DMZs can get pulled into the Medium Path for inspection. If you are using object Internet in your APCL policy, make sure that your firewall's interface topology definitions are complete and correct so that the dynamic object "Internet" is properly calculated.