VSX Connectivity Upgrade Error - CPHAPROB can't see old member on VSID 0.

[eduardoxmunoz]

Hi,

Hopefully someone can give me light on this issue...

scenario: VSX VSLS upgrade R77.10 to R77.30 Jumbo HFA 205 using connectivity upgrade method. All VS'es running on one member during the upgrade. VSX gateways being managed with Multi-Domain

Status before issuing 'cphacu start'

[Expert@VSX2:0]# cphaprob state

Cluster Mode:   Virtual System Load Sharing

Number     Unique Address  Assigned Load   State

2 (local)  10.255.255.66  0%              Ready

(*) 'Ready' state might be caused due to configuration inconsistency between members:
    32bit/64bit/usermode, number of CoreXL instances or different SW version.

Cluster name: FWVSX

Virtual Devices Status on each Cluster Member
=============================================

 ID    | Weight| FWVSX2
       |       | [local]
-------+-------+-----------
 1     | 10    | Ready
 2     | 10    | Ready
 3     | 10    | Ready
 4     | 10    | Ready
 5     | 10    | Ready
 6     | 10    | Ready
---------------+-----------
 Active        | 0
 Weight        | 0
 Weight (%)    | 0

Legend:  Init - Initializing, Active! - Active Attention
         Down! - ClusterXL Inactive or Virtual System is Down

'cphacu start' finishes with the following error: get_old_member_info: Connectivity upgrade failed since CPHAPROB can't see old member on VSID 0.

[Expert@VSX2:0]# cphacu start

Starting Connectivity Upgrade...

Waiting until the new member is able to communicate with Active old member... 1/10
Waiting until the new member is able to communicate with Active old member... 2/10
Waiting until the new member is able to communicate with Active old member... 3/10
Waiting until the new member is able to communicate with Active old member... 4/10
Waiting until the new member is able to communicate with Active old member... 5/10
Waiting until the new member is able to communicate with Active old member... 6/10
Waiting until the new member is able to communicate with Active old member... 7/10
Waiting until the new member is able to communicate with Active old member... 8/10
Waiting until the new member is able to communicate with Active old member... 9/10
Waiting until the new member is able to communicate with Active old member... 10/10
get_old_member_info: Connectivity upgrade failed since CPHAPROB can't see old member on VSID 0.

Traffic is impacted over the VS'es running in the Active VSX after 'cphacu start' finishes

Status in active gateway is

[Expert@VSX1:0]# cphaprob state

Cluster Mode:   Virtual System Load Sharing

Number     Unique Address  Assigned Load   State

1 (local)  10.255.255.65  100%            Active
2          10.255.255.66  0%              ClusterXL Inactive or Machine is Down

Cluster name: FWVSX

Virtual Devices Status on each Cluster Member
=============================================

 ID    | Weight| FWVSX1    | FWVSX2
       |       | [local]   |
-------+-------+-----------+-----------
 1     | 10    | Active!   | Down!
 2     | 10    | Active!   | Down!
 3     | 10    | Active!   | Down!
 4     | 10    | Active!   | Down!
 5     | 10    | Active!   | Down!
 6     | 10    | Active!   | Down!
---------------+-----------+-----------
 Active        | 6         | 0
 Weight        | 60        | 0
 Weight (%)    | 100       | 0

Legend:  Init - Initializing, Active! - Active Attention
         Down! - ClusterXL Inactive or Virtual System is Down

Status in upgraded gateway after 'cphacu start' is finished

[Expert@VSX2:0]# cphaprob state

Cluster Mode:   Virtual System Load Sharing

Number     Unique Address  Assigned Load   State

1          10.255.255.6  0%              ClusterXL Inactive or Machine is Down
2 (local)  10.255.255.66  0%              Ready

(*) 'Ready' state might be caused due to configuration inconsistency between members:
    32bit/64bit/usermode, number of CoreXL instances or different SW version.

Cluster name: FWVSX

Virtual Devices Status on each Cluster Member
=============================================

 ID    | Weight| FWVSX1    |  FWVSX2
       |       |           | [local]
-------+-------+-----------+-----------
 1     | 10    | Down!     | Ready
 2     | 10    | Down!     | Ready
 3     | 10    | Down!     | Ready
 4     | 10    | Down!     | Ready
 5     | 10    | Down!     | Ready
 6     | 10    | Down!     | Ready
---------------+-----------+-----------
 Active        | 0         | 0
 Weight        | 0         | 0
 Weight (%)    | 0         | 0

Legend:  Init - Initializing, Active! - Active Attention
         Down! - ClusterXL Inactive or Virtual System is Down

I have managed to upgrade this cluster (with a small outage). However, additional upgrades are planned over the following weeks and having minimal outage is a must.

Has anyone faced this issue?

The Connectivity Upgrade document describes a similar error with this description

When CU starts, the two members begin to communicate, and the new member sees the old member as Active. Check communication on the sync interface, and make sure that the MAC Magic Configuration is correct.

I have checked and:

- Magic MAC configuration seems to be the same in both gateways (even though the upgraded VSX makes use of the cluster_id parameter)
- SYNC interfaces are communicating properly

Thanks for any idea on this issue..

Regards

Ed

[varera]

Did you put them into broadcast mode before going for CU?

Here it seems they can hear each other but cannot sync.

[eduardoxmunoz]

Did you put them into broadcast mode before going for CU?

Hi Valeri,

No I didn't. However, I have been able to replicate the issue in my lab.

Thanks for the idea, I'll try it in my lab and come back with the results

Cheers

[varera]

looking forward to hear the findings.

[pawankiet1982]

Hi,

I have recently upgrade from R77.20 to R77.30 with hotfix take 216 and bulid 1283...

Once one standby node will upgrade need to run cphacu start CMD to sync connection table with another member..

Once sync done use cpstop on another node traffic move newly upgraded node...some intermittent traffic might be effected....

Till both node are not on same version it's shows ...ready /down.

Traffic going on ready ..

Sent from my A1601 using Tapatalk

[jonne]

Hi Ed,

I am preparing for a VSX upgrade from R77.10 to R77.30 and I have run into the same issue in my lab. I tried changing to broadcast mode, but that did not help. Did you find a solution?

Thanks,

Jonne.

[jonne]

Hi all,

I logged a support call about this issue and after some time, the engineer suggested to try a clean install instead of in-place upgrade. After doing a clean install, the Connectivity Upgrade worked. It appears that Connectivity Upgrade will only work with a clean install and not an in-place upgrade.

Thanks,

Jonne.

[eduardoxmunoz]

Hi Jonne,

Thanks for the updates. Unfortunately, I wasn't able to find a solution either. I managed to get approval for a short maintenance window :) and upgraded without using CU.

Cheers