WSO2 is now Choreo LabsLearn more

Trusted AI Governance
Build

Create agents and AI apps with the tools and languages your teams already use.

Connect

Link agents securely to your APIs, data, models and business systems.

Run

Deploy, observe and operate agents and AI apps reliably at any scale.

Govern

Control identity, access, cost and policy for every agent in production.

Agent Management

Developer Platform

  • API Management Cloud

    Design, secure and publish APIs with governance and analytics built in.

    Sign Up Now
  • Integration Cloud

    Connect apps, data and AI agents with low-code and pro-code integration.

    Sign Up Now
  • Identity & Access Management Cloud

    Secure sign-in and access for customers, partners, workforce and agents.

    Sign Up Now
  • Agent Management Cloud

    Build, run and govern AI agents with full visibility and control.

    Sign Up Now
  • Developer Platform Cloud

    Ship cloud-native apps faster on an internal developer platform.

    Sign Up Now

Self-Hosted

Full control over your stack. Deploy directly to your own servers, bare metal, cloud, or Kubernetes environment. Your data never leaves your perimeter.

Product Downloads
Choreo Labs
AI Governance
Platform
Solutions
Resources
Article

[Article] WSO2 API Manager deployment customization and troubleshooting - part 2

Introduction

WSO2 API Manager is a complete solution for publishing APIs, creating and managing a developer community, and for scalably routing API traffic. It leverages proven, production-ready, integration, security, and governance components from WSO2 Enterprise Service Bus, WSO2 Identity Server, and WSO2 Governance Registry. Moreover, it is powered by WSO2 Business Activity Monitor, thereby making WSO2 API Manager ready for any large-scale deployments right away.

When you work on WSO2 API Manager deployments you need to follow deployment best practices. Generally, with API management platform deployment, we need to focus on some customization that's specific to your use case. Moreover, you may also need to fine-tune some parameters according to specific requirements and system load. When you configure deployment, you may come across certain issues, and when deployment is up and running you may get some run-time issues due to various reasons. If you have some knowledge about the product and carbon platform, you might be able to sort out these issues fairly easily.

Applies to

WSO2 API Manager Version 1.8.0 and above

Table of contents

  1. Stats and usage data-related issues
  2. Distributed deployment-specific issues
  3. API invocation-related issues
  4. Other common issues
  5. Conclusion

Distributed deployment-specific issues

Connectivity between components in distributed deployment

We need to understand connectivity between components in distributed API manager deployment clearly.
This is important when you troubleshoot issues related to distributed API Manager deployment. Refer to the following steps to understand connectivity between components.

  • Changed the admin password
  • Tried to log in to publisher and got an insufficient privilege error.
  • Then changed the admin password in authManager element in api-manager.xml
  • Restarted and was able to login to API publisher.
  • In the API manager store and publisher, we need to specify auth manager details properly. In distributed deployment, we normally point to the key manager as the auth server. When the user tries to login, the system publisher/store will connect to the auth server to authenticate the user.
  • Then, we created an API and tried to publish. A permission error occurs again.
  • Then, we changed the password under the API gateway element in api-manager.xml
  • Restarted and published the API.
  • In the API publisher node, we need to configure the gateway URL and credentials properly. This information will be used to call admin services deployed in the gateway. When we create an API from the publisher and try to publish it to the gateway, we do a web service call to push API configuration in the gateway node.
  • Then, we tried to invoke an API using an existing key.
  • Got the key validation error
  • We then changed the admin password in KeyManager element in api-manager.xml and all issues got resolved.
  • Each and every request that comes to the gateway needs to pass the API authentication handler. In this handler, we will extract the required information and call the key validation service in the key manager server. For this, we need to configure the key manager server URL and credentials properly in api-manager.xml file in the gateway node.

Why thrift key validation is not recommend in production deployments

Thrift key validation does not work when we have a load balancer fronted key manager.
The reason for this is that most load balancers are not capable of routing traffic in a session-aware manner.
Therefore, in such cases, it's always recommended to use the WS key validation client. When we use web service clients we need to enable a sticky session at the load balancer level. Moreover, we need to give a session cookie that we've used to handle session stickiness.

Accessing APIs by single user using multiple devices/clients

As you may know, we allow one user to have one access token (for a given application) at a given time. But when we use multiple devices, the token may be revoked by one application. Therefore, users may request to have multiple active access tokens at a given time. However, we designed WSO2 API Manager to have one active token for application user combination. Hence, these types of requirements should be addressed by modifying the client (end-user application) according to custom requirements. This article [https://sanjeewamalalgoda.blogspot.com/2014/03/how-to-configure-wso2-api-manager-to.html] explains how you can achieve this.

What if the server goes OOM or starts spinning

In the event a server goes OOM, you would need to get a heap dump. It is normally dumped by default, if not, we have to dump manually. To find CPU spinning issues, get the thread dump and thread usage and try to find the culprit thread. Refer to this article [https://nirmalfdo.blogspot.com/2013/05/how-to-find-culprit-when-cpu-starts-to.html] for instructions.

Why APIs are not visible in the store in distributed deployment

Sometimes users may complain that they cannot see APIs in the store UI once they publish it from publisher. The API store and publisher will share the same registry. If the registry cache is not synchronized properly, then you will see some inconsistencies. Refer to the following instructions if you encounter such an issue.

  • Change the admin password
  • Store, Publisher should be in the same clustering domain
  • Cache ID should be correctly specified in registry.xml
  • If configuration is done correctly then, users can log into the management console and see if the APIs are indexed correctly
  • Registry indexing changes can cause to this issue

Why some changes made to APIs are not immediately reflected in the gateway

On the API gateway side, the API resource cache will have API metadata. If that's enabled, the changes on Auth type, and the addition of resources and changes in resource level throttling policies will not take effect until a cache expiration occurs. The default cache expiration is 15 minutes and you can try the same after that time limit.

Move API manager deployment between environments

Another common issue found in deployments is how to move one API manager deployment to another.
As of now, we don't have a one-step migration process. However, it can be done using the following steps listed here [https://sanjeewamalalgoda.blogspot.com/2014/03/how-to-move-wso2-api-management.html]
We have planned to introduce deployable API artifacts for WSO2 API Manager in the future.

Other common issues

When we have public deployment how we can let users tryout APIs with the global key concept?

When users opt for a public API store, they need to let their users try the API without subscribing to it (i.e. without getting an access token). If you ask them to login, create an application, subscribe, obtain a token, and use it, they would need to spend a significant amount of time to invoke the API. Therefore, it would be better to allow them to invoke APIs with one click, even without login. However, you need to apply the correct throttling to avoid misuse of APIs.
This capability is available in most of publicly available API stores. This [https://sanjeewamalalgoda.blogspot.com/2014/03/how-to-customize-wso2-api-manager.html] article describes how you can customize our API manager to address that requirement.

Deleting APIs

We noticed that some users try to remove APIs by deleting registry API artifact + synapse config and database entry. This is a wrong move. We should always ask users to delete APIs using the publisher user interface. If you delete the API part by part (from registry, database, and synapse) then you will get multiple issues due to active subscriptions, resource mapping, and other dependencies.
Sometimes when you delete registry artifacts, API artifacts may get deleted due to user errors. If we delete API artifact from the registry, it will cause issues in the API store and publisher. When we load APIs in store/publisher applications, we normally retrieve resources from registry. If API data in the registry gets deleted due to some reason, we can fix it by following the instructions listed here [https://sanjeewamalalgoda.blogspot.com/2013/11/how-to-fix-issue-in-subscription-page.html].

Multiple level throttling in API manager

Another common issue is tuning parameters and minimum deployment requirements. We can use the instructions listed in these articles [https://docs.wso2.org/display/AM150/Performance+Tuning+Recommendations][https://sanjeewamalalgoda.blogspot.com/2013/11/recommended-system-requirement-for-wso2.html] when we setup deployment. Moreover, we can apply ESB tuning parameters to the API gateway as it's almost similar to ESB.

Minimum hardware requirements and tuning parameters

The API manager has throttling in 3 levels. We introduced resource level throttling in WSO2 API Manager 1.5.0; here [https://sanjeewamalalgoda.blogspot.com/2013/09/throttling-support-at-api-resource.html] is a brief description on how it works.

Writing test cases

From WSO2 API Manager 1.7.0 onwards, you will be able to write Java test cases easily as we expose common API management operations as Java utility methods (you can find the store and publisher rest client source code here [[12]https://svn.wso2.org/repos/wso2/carbon/platform/branches/turing/products/apimgt/1.7.0/modules/integration/tests/src/test/java/org/wso2/carbon/am/tests/util/]). We will also be able to add host object functionalities for automated tests (refer to this sample tests [https://svn.wso2.org/repos/wso2/carbon/platform/branches/turing/products/apimgt/1.7.0/modules/integration/tests/src/test/java/org/wso2/carbon/am/tests/sample/HostObjectTestCase.java]). These kinds of integration tests are important because you don't have to test all functionality manually once you make some change to the system.

Conclusion

In this article series, we discussed the basics of WSO2 API Manager deployments. This part of the two-part article mainly focused on deployment troubleshooting. We discussed deployment best practices and fine tuning parameters as well. When you deploy production deployment, the minimum down time is a key element that you would need to pay attention to. With this, you should be able to sort out any issues that crop up within a short period of time.

References

[2] https://docs.wso2.com/display/AM180/WSO2+API+Manager+Documentation