established SIC between the CMA and enforcement modules

[cciesec2006]

I have a situation that needs urgent attention. I can not
figure out as to why it behaves this way:

SPLAT is 2.4 kernel

1) I have a gateway gwA and gwB as enforcement modules running
SPLAT NG with AI R55 with HFA_17 on Sun iForce platform with
dual processors and 512MB RAM. gwA and gwB are single
enforcement modules being managed by a CMA inside Provider-1
Secureplatform NGx R65 with HFA_02 and hf_249. Everything
is working fine so far. I am running QoS on gwA but NO QoS on gwB.
in "sysconfig", I configured both gwA and gwB as VPN-1 Power.
I have VALID on the CMA SmartUpdate.

Today, I have to upgrade gwA and gwB to NGx R65 with HFA_02 and
hf_249. I built both gwA and gwB gateways in the with identical
IP addresses as the one in the production environment. The hardware
of gwA and gwB is IBM x3650 with dual quad-core processors and
4GB of RAM. I built both gwA and gwB in the lab and make sure
all IP addresses match the one in the production environment.
I then set the SIC key to "cisco123" on both gwA and gwB.
I also make sure that both gwA and gwB are synced the clock
with an NTP server in my lab so the clock is very accurate.
Both the P-1 and gwA/gwB are off by about 0.000002 secconds.

I then moved the NGx gwA and gwB in to the production environment.

I then shutdown both NG with AI gwA and gwB and powered on the
NGx gwA and gwB. THERE ARE NO ADDITIONAL FIREWALLS BETWEEN THE CMA
AND gwA AND gwB. From console, I then performed "fw unloadlocal" on
both gwA and gwB. From the gwA and gwB console, I then type
"ntpdate 4.2.2.2" where 4.2.2.2 is the NTP server on my network.
I checked again, the P-1 box and gwA/gwB are off by about 0.000003 sec.

From the CMA, for gwB gateway object, I change the software version
to NGx R65 VPN-1 Power, Secureplatform, then I re-SIC gwB in the
CMA. The SIC went OK. After that, I used SmartUpdate to attach
the NGx license to gwB. Finally, I pushed the policy and the policy
is successfully installed on gwB.

From the CMA, for gwA gateway object, I change the software version
to NGx R65 VPN-1 power, and Secureplatform. When I tried to re-SIC
gwA, from the CMA, it just sat there about 30 seconds, and I got this
message: "Internal SSL certificate error". When I run "cpconfig", on
gwA, it tells me that SIC is established BUT I can NOT push policy to
gwA. I tried to re-SIC with "cpconfig" on gwA with "cpstop;cpstart"
and "fw unloadlocal" after that. That did NOT work. I rebooted
gwA several times, then re-SIC several times on gwA but I keep getting
the same "SSL certificate error". I DO HAVE CONNECTIVITY FROM gwA
TO THE CMA. THERE ARE NO FIREWALLS IN BETWEEN gwA AND THE CMA. ALL THE
PORTS ARE WIDE OPEN.

Usually an error likes this points to out-of-sync between the CMA and
gwA but the clock on gwA and the CMA are about 0.000003 seconds with
one another. I am not sure if rebuilding gwA will fix this problem either.

Has anyone run into something like this before and how do you go about
fixing it?

Thanks in advance.

[Thorpuse]

I think the issue is on your CMA - sounds like the gateway certificate did not generate correctly. Check for duplicate objects using the same IP addresses as GWa. Also (if you can) stop and start the CMA, and have a look in the ICA logfiles and the ICA management tool for the status of the ICA.

[cciesec2006]

Originally Posted by Thorpuse

I think the issue is on your CMA - sounds like the gateway certificate did not generate correctly. Check for duplicate objects using the same IP addresses as GWa. Also (if you can) stop and start the CMA, and have a look in the ICA logfiles and the ICA management tool for the status of the ICA.

One thing I would like to add is that when I powered off NGx R65 gwA
and powered NG with AI R55 gwA, I have no issue re-SIC the CMA with
the R55 gwA gateway.

If the gateway certificate did not generate correctly, does that apply
to NG with AI R55 gwA gateway as well?

Thanks.

David

[Thorpuse]

Not sure. doesn't seem to make a lot of sense. You may want to do a reset, save/install policy to GWb (to regenerate the CRL), then try to establish trust to GWa. Anything in the logs on either side?

[msjouw]

is there any WAN accellaration device in between the CMA and the gateway? IF so make sure to disable it, we had a box like that do caching of the cert. Other thing to check is the duplex settings on switch and gwA.

[cciesec2006]

Originally Posted by msjouw

is there any WAN accellaration device in between the CMA and the gateway? IF so make sure to disable it, we had a box like that do caching of the cert. Other thing to check is the duplex settings on switch and gwA.

1- No WAAS box between CMA and gateway.
2- speed/duplex settings on the switch and gwA are set to auto/auto on both
sides because we are using copper Gig Ethernet on both the IBM and
Cisco Catalyst 6513 with Sup-720.

On point #2, I've looked at it many times and that I have another
CCIE look at the layer-2 configration (I myself also a CCIE) so everything is
setup correctly the same way we setup with gwB.