WSO2 has many products that can be combined flexibly to scale and meet business needs. There is extensive and useful documentation about clustering
here. However, this doesn't cover each and every scenario, because there are so many possible combinations. In this post, I will outline how to set up a distributed API Manager (AM) with the Identity Server (IS) as a Key Manager and a worker/manager Gateway cluster fronted by the Elastic Load Balancer (ELB). This setup also allows tenanting.
Architecture
Before I describe how to install or configure the products, I need to lay out the architecture I plan to use, defining nodes, roles, clusters, ports, databases, domain and host names.
For this exercise, I will set up all the products on a single machine running Linux Mint. Each product will each have a distinct port offset. Wherever possible, I will use host names instead of IPs to refer to other nodes. This requires some customization of /etc/hosts.
This configuration uses five distinct AM servers, in blue. The basic idea of API Manager separation into these roles is well documented
here. Adding an Identity Server as a Key Manger is documented
here. Worker-manager separation with a load balancer is covered
here. If you only need to apply one of these concepts, please follow the existing documentation. Otherwise, please continue!
Versions
I'm using the Carbon 4.2.0 platform, with
WSO2 AM 1.7.0
WSO2 IS 5.0.0
WSO2 ELB 2.1.1
HTTP Ports and Offsets
Each instance installed on a single machine must have a distinct port offset. This creates some complications, so I want to be very clear with the port definitions up-front. First of all, there are four HTTP ports for each AM instance, including management HTTP/HTTPS ports and non-blocking HTTP-NIO (also called NHTTP and NHTTPS) ports. To learn about the non-blocking NIO technology, you can read more on the
Apache site or in the WSO2
documentation. For more information on WSO2 products and ports, see the documentation
here.
Clustering Ports
Clustering ports are distinct from the ports used for http. Note that the cluster ports here have nothing to do with the http port
offsets. Also, the Store and Publisher are in a cluster together, as
shown by the domain. I'm not setting up clustering for the Identity
server here. I'm also not using the ELB to front the Store or
Publisher.
Host Names
I will use host names for individual servers, and there are also host names used by the ELB for the clustered groups (worker and manager). The red arrows below define how the host name is used to pass messages through the load balancer. These host names will need to be added to /etc/conf. Please note that the host names for the load-balanced groups will need
to route to the load balancer via etc/conf or another means, and the
load balancer will forward these on to the clustered group members.
Databases
Instead of using only the default H2/Cassandra databases, I'm using a local MySQL db. There are three databases here, which are referred to by either a jndi Config name, host name and/or the MySQL name:
| MySQL DB Name |
jndi Config Name |
Host Name |
| userstore |
jdbc/WSO2UM_DB |
userdb.mysql-wso2.com |
| registry |
jdbc/WSO2REG_DB |
regdb.mysql-wso2.com |
| apimgt |
jdbc/WSO2AM_DB |
apimgtdb.mysql-wso2.com |
The apimgt DB can be connected to the Store, Publisher and IS only, without connecting to the gateway nodes. The other two DBs can be connected to all the nodes except the ELB. In some security situations, you may wish to further restrict which nodes have access to which databases, but I'll keep it a little simple here. In production settings where the gateway nodes are outside the firewall, a different configuration would be necessary.
Database Connectivity
Installation
In this case, I will create a folder structure like /home/username/wso2/AM_IS_ELB/ and create 7 sub-folder directories, one for each server I'm installing. You can name these whatever you like. Download the source from WSO2 and extract the zipped contents to the relevant directories. I will refer to these directories like this through the rest of this post: <Load Balancer Home>.
Configuration
carbon.xml
Open /repository/conf/carbon.xml on for the Gateway Manager, Gateway workers, Store, Publisher and Identity Server. Change the port offset for each node.
1. Gateway Manager, <Offset>1</Offset>
2. Gateway Worker1, <Offset>2</Offset>
3. Gateway Worker2, <Offset3</Offset>
4. Store, <Offset>4</Offset>
5. Publisher, <Offset>5</Offset>
6. IS, <Offset>10</Offset>
7. ELB, no change necessary.
Adding Features to the Identity Server
Before continuing with configuration, we need to add a few additional features to the Identity server. This is outlined well in the
WSO2 documentation, from steps 3.a to 3.c.iii. . Don't continue to step 3.c.iv .
Database Setup
For setting up the databases, please refer to steps 1 through 3 in the 'Installing and configuring the Databases'
here. Then, follow steps 8
and 9
here. Don't forget to copy over the MySQL jdbc driver to each node.
master-datasources.xml
Open /repository/conf/datasources/master-datasources.xml for the Gateway Manager, Gateway workers, Store, Publisher and Identity Server. For the Gateway Manager, Gateway Worker1 and Gateway Worker2, add the following two datasources:
<datasource>
<name>WSO2REG_DB</name>
<description>The datasource used for registry and user manager</description>
<jndiConfig>
<name>jdbc/WSO2REG_DB</name>
</jndiConfig>
<definition type="RDBMS">
<configuration>
<url>jdbc:mysql://regdb.mysql-wso2.com:3306/registry?autoReconnect=true&relaxAutoCommit=true&</url>
<username>apiuser</username>
<password>apimanager</password>
<driverClassName>com.mysql.jdbc.Driver</driverClassName>
<maxActive>50</maxActive>
<maxWait>60000</maxWait>
<testOnBorrow>true</testOnBorrow>
<validationQuery>SELECT 1</validationQuery>
<validationInterval>30000</validationInterval>
</configuration>
</definition>
</datasource>
<datasource>
<name>WSO2UM_DB</name>
<description>The datasource used for registry and user manager</description>
<jndiConfig>
<name>jdbc/WSO2UM_DB</name>
</jndiConfig>
<definition type="RDBMS">
<configuration>
<url>jdbc:mysql://userdb.mysql-wso2.com:3306/userstore?autoReconnect=true&relaxAutoCommit=true&
</url>
<username>apiuser</username>
<password>apimanager</password>
<driverClassName>com.mysql.jdbc.Driver</driverClassName>
<maxActive>50</maxActive>
<maxWait>60000</maxWait>
<testOnBorrow>true</testOnBorrow>
<validationQuery>SELECT 1</validationQuery>
<validationInterval>30000</validationInterval>
</configuration>
</definition>
</datasource>
For the Store, Publisher and Identity Server master-datasources.xml, first add the datasources above. Then comment out the existing entry for WSO2AM_DB and replace it with the new entry for WSO2AM_DB below.
<datasource><name>WSO2AM_DB</name>
<description>The datasource used for API Manager database</description>
<jndiConfig>
<name>jdbc/WSO2AM_DB</name>
</jndiConfig>
<definition type="RDBMS">
<configuration>
<url>jdbc:mysql://apimgtdb.mysql-wso2.com:3306/apimgt?autoReconnect=true&relaxAutoCommit=true&</url>
<username>apiuser</username>
<password>apimanager</password>
<driverClassName>com.mysql.jdbc.Driver</driverClassName>
<maxActive>50</maxActive>
<maxWait>60000</maxWait>
<testOnBorrow>true</testOnBorrow>
<validationQuery>SELECT 1</validationQuery>
<validationInterval>30000</validationInterval>
</configuration> </definition>
< </datasource>
Now that the datasources are set up, we can use the jndiConfig name in other property files to connect to the databases.
user-mgt.xml
Open the /repository/conf/user-mgt.xml for the Gateway Manager, Gateway
workers, Store, Publisher and Identity Server. Change the datasource name to WSO2UM_DB:
<Realm>
...
<Configuration>
...
<Property name="dataSource">jdbc/WSO2UM_DB</Property>
...
</Configuration>
...
</Realm>
registry.xml
Copy the
registry.xml file from the
<GATEWAY_MANAGER>/repository/conf directory to the
<IS_HOME>/repository/conf directory. Make sure you replace the existing
registry.xml file found in the
<IS_HOME>. This is because the
<indexingConfiguration> element is not there in the
registry.xml that comes in the IS.
Open the
/repository/conf/registry.xml for the Gateway
Manager, Gateway
workers, Store, Publisher, and Identity Server. In each file, add this registry configuration:
<dbConfig name="govregistry">
<dataSource>jdbc/WSO2REG_DB</dataSource>
</dbConfig>
<remoteInstance url="https://localhost">
<id>gov</id>
<dbConfig>govregistry</dbConfig>
<readOnly>false</readOnly>
<enableCache>true</enableCache>
<registryRoot>/</registryRoot>
</remoteInstance>
<mount path="/_system/governance" overwrite="true">
<instanceId>gov</instanceId>
<targetPath>/_system/governance</targetPath>
</mount>
<mount path="/_system/config" overwrite="true">
<instanceId>gov</instanceId>
<targetPath>/_system/config</targetPath>
</mount>
In addition, add the following code block to the
<IS_HOME>/repository/conf/registry.xml :
<handler class="org.wso2.carbon.identity.entitlement.policy.finder.registry.RegistryPolicyHandler">
<filter class="org.wso2.carbon.identity.entitlement.policy.finder.registry.RegistryPolicyMediaTypeMatcher">
<property name="mediaType">application/xacml-policy+xml</property></filter> </handler>
identity.xml
Open the identity.xml file in <IS_HOME>/repository/conf/. Change the datasource from the default carbonDB to the mySql WSO2AM_DB:
<DataSource>
...
<Name>jdbc/WSO2AM_DB</Name>
...
</DataSource>
application-authentication.xml
Open the identity.xml file in <IS_HOME>/repository/conf/security/application-authentication.xml. Change the datasource from the default carbonDB to the mySql WSO2AM_DB:
<DataSource>
</handler>
...
<Name>jdbc/WSO2AM_DB</Name>
...
</DataSource>
loadbalancer.conf
Open the <ELB_HOME>/repository/conf/loadbalancer.conf. Add the following section:
am {
# multiple hosts should be separated by a comma.
hosts am.cloud-test.wso2.com;
#url_suffix am.wso2.com;
domains {
am.gateway.domain {
tenant_range *;
group_mgt_port 4500;
worker {
hosts am.wso2.com;
}
mgt {
hosts mgt.am.wso2.com;
}
}
}
}
api-manager.xml
Copy the api-manager.xml file from the <GATEWAY MANAGER HOME>/repository/conf/ directory to the <IS_HOME>/repository/conf/
directory. Make sure you replace the existing api-manager.xml file found in the <IS_HOME>. Open the newly copied <IS_HOME>/repository/conf/api-manager.xml and perform these edits:
1. Change the GatewayType property to the following.
<GatewayType>None</GatewayType>
2. Change the <RevokeAPIURL> so that it points to the API Gateway Workers via the load balancer.
<RevokeAPIURL>https://am.wso2.com:8243/revoke</RevokeAPIURL>
3. Change the <ServerURL> occurring under the <APIGateway> section so that it points to the API Gateway Manager via the load balancer.
<ServerURL>https://mgt.am.wso2.com:443/services/</ServerURL>
4. Change EnableThriftServer to false.
<EnableThriftServer>false</EnableThriftServer>
Open the /repository/conf/api-manager.xml for the Gateway Manager, Gateway Workers, Store, and Publisher. Perform the these edits:
1. Change the ServerURL of the AuthManager to point to IS.
<ServerURL>https://is.wso2.com:9453/services/</ServerURL>
2. Change the ServerURL of the APIKeyManager to point to IS.
<ServerURL>https://is.wso2.com:9453/services/</ServerURL>
3. Change the KeyValidatorClientType from ThriftClient to WSClient.
4. Change EnableThriftServer to false.
In the Store and Publisher api-manager.xml, connect to the Gateway Manager node via the ELB. Change the following URLs and Endpoints:
<APIGateway>
<Environments>
<Environment type="hybrid">
...
<ServerURL>https://mgt.am.wso2.com:9445/services/</ServerURL>
...
<GatewayEndpoint>http://mgt.am.wso2.com:8280,https://mgt.am.wso2.com:8243</GatewayEndpoint>
</Environment>
</Environments>
...
</APIGateway>
axis2.xml
Open the <Load Balancer Home>/repository/conf/axis2/axis2.xml . Locate the Transport Ins (Listeners) section and change the PassThroughHttpListner to the default http ports as follows:
<transportReceiver name="http" class="org.apache.synapse.transport.passthru.PassThroughHttpListener">
...
<parameter name="port">80</parameter>
...
</transportReceiver>
<transportReceiver name="https" class="org.apache.synapse.transport.passthru.PassThroughHttpSSLListener">
...
<parameter name="port" locked="false">443</parameter>
...
</transportReceiver>
Please note that the above changes to the default http/s listener ports will require that the Load Balancer be started with administrative access, using sudo or su.
Still in the same Load Balancer's axis2.xml file, in the Transport Ins (Listeners) section, add additional transports for 8280 and 8243 ports:
<transportReceiver name="http" class="org.wso2.carbon.core.transports.http.HttpTransportListener">
<parameter name="port">8280</parameter>
</transportReceiver>
<transportReceiver name="https" class="org.wso2.carbon.core.transports.http.HttpsTransportListener">
<parameter name="port">8243</parameter><parameter name="keystore" locked="false">
<KeyStore>
<Location>repository/resources/security/wso2carbon.jks</Location>
<Type>JKS</Type>
<Password>wso2carbon</Password>
<KeyPassword>wso2carbon</KeyPassword>
</KeyStore>
</parameter>
<parameter name="truststore" locked="false">
<TrustStore>
<Location>repository/resources/security/client-truststore.jks</Location>
<Type>JKS</Type>
<Password>wso2carbon</Password>
</TrustStore>
</parameter>
</transportReceiver>
tory/conf/axis2/axis2.xml for the Gateway
Manager, Gateway
workers, Store and Publisher. Each file will be configured differently:
1. Change the HazelcastClusteringAgent to enable="true"
<clustering class="org.wso2.carbon.core.clustering.hazelcast.HazelcastClusteringAgent" enable="true">
2. Change the membershipScheme to wka
<parameter name="membershipScheme">wka</parameter>
3. Change the domain to the appropriate clustering domain
a. For the gateway manager and workers:
<parameter name="domain">am.gateway.domain</parameter>
b. For the Store and Publisher:
<parameter name="domain">wso2.pub.store.domain</parameter>
4. Change the local member port to the pre-determined port, which is different for each node:
<parameter name="localMemberPort">xxxx</parameter>
a. Gateway Manager: 4001
b. Gateway Worker1: 4002
c. Gateway Worker2: 4003
d. Store: 8011
e. Publisher: 8010
5. Only for the Gateway Manager node, in the <Parameter name="Properties"> section, change the sub-domain to mgt
<property name="subDomain" value="mgt"/>
6. Add these Port Mappings to each node except the IS and ELB. They should be added in the <Parameter name="Properties"> section:
a. Gateway Manager:
<property name="port.mapping.80" value="9764"/>
<property name="port.mapping.443" value="9444"/>
b. Gateway Worker1:
<property name="port.mapping.80" value="9765"/>
<property name="port.mapping.443" value="9445"/>
c. Gateway Worker2:
<property name="port.mapping.80" value="9766"/>
<property name="port.mapping.443" value="9446"/>
d. Store:
<property name="port.mapping.80" value="9767"/>
<property name="port.mapping.443" value="9447"/>
e. Publisher:
<property name="port.mapping.80" value="9768"/>
<property name="port.mapping.443" value="9448"/>
7. Replace the default <members> section with correct well-known members. These include the cluster partners' IPs and clustering ports. Please note that this is the one place in this setup guide where it is necessary to use the actual IP of the servers. This isn't a limitation of the WSO2 software, but it is an issue in the HazelCast cluster framework we use. It should be fixed in the next versio<!-- note the use of the loopback IP -->n of HazelCast that WSO2 implements with the next version of Carbon. Sorry for the inconvenience, but you should use your local IP here in place of xxx.xxx.xxx.xxx, and not just localhost or other host names. Please use ifconfig or a similar tool to get your ip.
a. Gateway Manager:
<members>
<member>
<hostName>xxx.xxx.xxx.xxx</hostName>
<port>4500</port>
</member>
<member>
<hostName>xxx.xxx.xxx.xxx</hostName>
<port>4002</port>
</member>
<member>
<hostName>xxx.xxx.xxx.xxx</hostName>
<port>4003</port>
</member>
</members>
b. Gateway Worker1:
<members>
<member>
<hostName>xxx.xxx.xxx.xxx</hostName>
<port>4500</port>
</member>
<member>
<hostName>xxx.xxx.xxx.xxx</hostName>
<port>4001</port>
</member>
<member>
<hostName>xxx.xxx.xxx.xxx</hostName>
<port>4003</port>
</member>
</members>
<address uri="https://is.wso2.com:9453/oauth2/authorize"/>
c. Gateway Worker2:
<members>
<member>
<hostName>xxx.xxx.xxx.xxx</hostName>
<port>4500</port>
</member>
<member>
<hostName>xxx.xxx.xxx.xxx</hostName>
<port>4001</port>
Update App URLs </member>
<member>
<hostName>xxx.xxx.xxx.xxx</hostName>
<port>4002</port>
</member>
</members>
d. Store:
<members>
<member>
<hostName>xxx.xxx.xxx.xxx</hostName>
<port>8010</port>
</member>
</members>
e. Publisher:
<members>
<member>
<hostName>xxx.xxx.xxx.xxx</hostName>
<port>8011</port>
</member>
</members>
etc/hosts
Open the etc/hosts at the machine root, with administrative privileges. Add the following entries. Please note, that this configuration would be different if you were installing these nodes on different physical or virtual machines, but here they are all on the same local machine.
127.0.0.1 elb.wso2.com mgt.am.wso2.com am.wso2.com gw.mgt.wso2.com store.wso2.com gw.worker1.wso2.com gw.worker2.wso2.com pub.wso2.com is.wso2.com userdb.mysql-wso2.com regdb.mysql-wso2.com apimgtdb.mysql-wso2.com am.cloud-test.wso2.com
SVN Deployment Synchronizer
It is necessary to configure subversion in order to sync deployment resources between the Gateway Manager and Workers. This is a complex process, but there is no difference between this installation and other worker/manager clusters of WSO2 products. Please follow the documentation
here to set up the SVN Deployment Synchronizer.
Update App URLs
It is necessary to update the app urls in <gateway manager home>repository/deployment/server/synapse-configs/default/api/. There are three urls here which need to be udpated: _TokenAPI_.xml, _RevokeAPI_.xml and _AuthorizeAPI_.xml. These URLs should point to the key manager.
For the TokenAPI:
<address uri="https://is.wso2.com:9453/oauth2/token"/>
For the RevokeAPI:
<address uri="https://is.wso2.com:9453/oauth2/revoke"/>
For the AuthorizeAPI:
<address uri="https://is.wso2.com:9453/oauth2/authorize"/>
Starting Up
Start up each server by navigating to the /bin directory and executing the wso2server.sh command. For the ELB, you will need to execute this command as an administrator. For the Gateway Worker nodes, please execute the wso2server.sh with the -DWorkerNode=true command.
sh wso2server.sh -DworkerNode=true
Testing
Start up a backend server of your choice (I usually use a standalone WSO2 DSS for testing). Use the publisher node at https://store.wso2.com:9448/publisher to publish an API. Use the Store at https://store.wso2.com:9447/store to subscribe to the API. Use a stand-alone client of your choice (postman, curl, or SoapUI are my favorites) to perform the oAuth2 and test your API, using the load balanced gateway worker url: https:/am.wso2.com/.
Best of luck. Please post your results, questions or any comments that could improve this post. Cheers, y'all!
