WSO2 is now Choreo LabsLearn more

Trusted AI Governance
Build

Create agents and AI apps with the tools and languages your teams already use.

Connect

Link agents securely to your APIs, data, models and business systems.

Run

Deploy, observe and operate agents and AI apps reliably at any scale.

Govern

Control identity, access, cost and policy for every agent in production.

Agent Management

Developer Platform

  • API Management Cloud

    Design, secure and publish APIs with governance and analytics built in.

    Sign Up Now
  • Integration Cloud

    Connect apps, data and AI agents with low-code and pro-code integration.

    Sign Up Now
  • Identity & Access Management Cloud

    Secure sign-in and access for customers, partners, workforce and agents.

    Sign Up Now
  • Agent Management Cloud

    Build, run and govern AI agents with full visibility and control.

    Sign Up Now
  • Developer Platform Cloud

    Ship cloud-native apps faster on an internal developer platform.

    Sign Up Now

Self-Hosted

Full control over your stack. Deploy directly to your own servers, bare metal, cloud, or Kubernetes environment. Your data never leaves your perimeter.

Product Downloads
Choreo Labs
AI Governance
Platform
Solutions
Resources
Article

Troubleshooting WSO2 product clusters

Introduction

If you are using Registry separation implementation and sharing configuration and governance registries among cluster nodes of any WSO2 product, this document will be useful. Normally, this is an issue users encounter during cluster restart after applying some change to the cluster. As an example, you might have done some change in product instances like applying a patch or any configuration change and then tried restarting the cluster nodes and the central Governance Registry.

Applies To

Product Version
WSO2 G-REG 3.*.*
WSO2 ESB 3.0.*

 

 

Troubleshooting

During the restart process, if you see that all the nodes are hangs with the below logs, please follow the given steps to fix it.

[2010-12-07 16:42:03,778]  INFO -  Initializing system...
[2010-12-07 16:42:09,659]  INFO -  Starting WSO2 Carbon...
[2010-12-07 16:42:09,660]  INFO -  Operating System : Linux 2.6.18-194.11.1.el5, amd64
[2010-12-07 16:42:09,660]  INFO -  Java Home        : /usr/lib/jvm/java-1.6.0-sun-1.6.0.7.x86_64/jre
[2010-12-07 16:42:09,660]  INFO -  Java Version     : 1.6.0_07
[2010-12-07 16:42:09,660]  INFO -  Java VM          : Java HotSpot(TM) 64-Bit Server VM 10.0-b23,Sun Microsystems Inc.
[2010-12-07 16:42:09,660]  INFO -  Carbon Home      : /home/kindrasi/wso2/wso2greg-3.5.1
[2010-12-07 16:42:09,660]  INFO -  Java Temp Dir    : /home/kindrasi/wso2/wso2greg-3.5.1/tmp
[2010-12-07 16:42:09,661]  INFO -  User             : root, en-US, America/New_York
 
So, obviously, users will highly doubt this problem causes the change they have made to the cluster. But before investigating based on that assumption, we recommend users do the following check.
 
Please access the database of the central WSO2 GReg database and select the values of the table REG_CLUSTER_LOCK. If you see the value of REG_LOCK_STATUS as LOCKED, we recommend you set the value to UNLOCKED and try to start the instances starting from the central Governance Registry instance.
 
This is due to an issue in products released based on Carbon 3.0.x. Root cause of the issue is setting the state LOCKED in the database table REG_CLUSTER_LOCK in Governance Registry database, during the restart process of cluster nodes. Since the products in the cluster is having mounts with central Governance Registry database, the start up of all the cluster nodes hangs.
 
From this test you can isolate the issue, if you still cannot start, this might be due to the change you have done to the cluster before restarting.
 

Author

Lahiru Gunathilaka

Senior Software Engineer; WSO2 Inc.

[email protected]