springboot+dubbo项目启动项目时报错 zookeeper not connected_caused by: java.lang.illegalstateexception: zookee

现象

项目在公司网络启动时，能正常启动。但通过vpn连接到公司网络时却无法启动报下面的错误java.lang.IllegalStateException: zookeeper not connected。

Caused by: java.lang.IllegalStateException: zookeeper not connected
	at org.apache.dubbo.remoting.zookeeper.curator.CuratorZookeeperClient.<init>(CuratorZookeeperClient.java:84)
	at org.apache.dubbo.remoting.zookeeper.curator.CuratorZookeeperTransporter.createZookeeperClient(CuratorZookeeperTransporter.java:26)
	at org.apache.dubbo.remoting.zookeeper.AbstractZookeeperTransporter.connect(AbstractZookeeperTransporter.java:68)
	at org.apache.dubbo.configcenter.support.zookeeper.ZookeeperDynamicConfiguration.<init>(ZookeeperDynamicConfiguration.java:66)
	at org.apache.dubbo.configcenter.support.zookeeper.ZookeeperDynamicConfigurationFactory.createDynamicConfiguration(ZookeeperDynamicConfigurationFactory.java:43)
	at org.apache.dubbo.common.config.configcenter.AbstractDynamicConfigurationFactory.lambda$getDynamicConfiguration$0(AbstractDynamicConfigurationFactory.java:39)
	at java.base/java.util.concurrent.ConcurrentHashMap.computeIfAbsent(ConcurrentHashMap.java:1705)
	at org.apache.dubbo.common.config.configcenter.AbstractDynamicConfigurationFactory.getDynamicConfiguration(AbstractDynamicConfigurationFactory.java:39)
	at org.apache.dubbo.common.config.configcenter.DynamicConfiguration.getDynamicConfiguration(DynamicConfiguration.java:224)
	at org.apache.dubbo.config.bootstrap.DubboBootstrap.prepareEnvironment(DubboBootstrap.java:1024)
	at org.apache.dubbo.config.bootstrap.DubboBootstrap.startConfigCenter(DubboBootstrap.java:619)
	at org.apache.dubbo.config.bootstrap.DubboBootstrap.initialize(DubboBootstrap.java:521)
	at org.apache.dubbo.config.ReferenceConfig.init(ReferenceConfig.java:244)
	at org.apache.dubbo.config.ReferenceConfig.get(ReferenceConfig.java:206)
	at org.apache.dubbo.config.spring.beans.factory.annotation.ReferenceAnnotationBeanPostProcessor.doGetInjectedBean(ReferenceAnnotationBeanPostProcessor.java:165)
	at com.alibaba.spring.beans.factory.annotation.AbstractAnnotationBeanPostProcessor.getInjectedObject(AbstractAnnotationBeanPostProcessor.java:409)
	at com.alibaba.spring.beans.factory.annotation.AbstractAnnotationBeanPostProcessor$AnnotatedFieldElement.inject(AbstractAnnotationBeanPostProcessor.java:626)
	at org.springframework.beans.factory.annotation.InjectionMetadata.inject(InjectionMetadata.java:119)
	at com.alibaba.spring.beans.factory.annotation.AbstractAnnotationBeanPostProcessor.postProcessPropertyValues(AbstractAnnotationBeanPostProcessor.java:179)
	... 78 common frames omitted
Caused by: java.lang.IllegalStateException: zookeeper not connected
	at org.apache.dubbo.remoting.zookeeper.curator.CuratorZookeeperClient.<init>(CuratorZookeeperClient.java:81)
	... 96 common frames omitted
Disconnected from the target VM, address: '127.0.0.1:52979', transport: 'socket'

Process finished with exit code 1
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27

猜测：连接到zookeeper的服务器网络策略不通
— 通过telnet证实端口网络时通的。该猜测不成立。

定位问题

进一步经过报错的日志，找到对应的源码。

public class CuratorZookeeperClient extends AbstractZookeeperClient<CuratorZookeeperClient.NodeCacheListenerImpl, CuratorZookeeperClient.CuratorWatcherImpl> {

    protected static final Logger logger = LoggerFactory.getLogger(CuratorZookeeperClient.class);
    private static final String ZK_SESSION_EXPIRE_KEY = "zk.session.expire";

    static final Charset CHARSET = StandardCharsets.UTF_8;
    private final CuratorFramework client;
    private static Map<String, NodeCache> nodeCacheMap = new ConcurrentHashMap<>();

    public CuratorZookeeperClient(URL url) {
        super(url);
        try {
            int timeout = url.getParameter(TIMEOUT_KEY, DEFAULT_CONNECTION_TIMEOUT_MS);
            int sessionExpireMs = url.getParameter(ZK_SESSION_EXPIRE_KEY, DEFAULT_SESSION_TIMEOUT_MS);
            CuratorFrameworkFactory.Builder builder = CuratorFrameworkFactory.builder()
                    .connectString(url.getBackupAddress())
                    .retryPolicy(new RetryNTimes(1, 1000))
                    .connectionTimeoutMs(timeout)
                    .sessionTimeoutMs(sessionExpireMs);
            String authority = url.getAuthority();
            if (authority != null && authority.length() > 0) {
                builder = builder.authorization("digest", authority.getBytes());
            }
            client = builder.build();
            client.getConnectionStateListenable().addListener(new CuratorConnectionStateListener(url));
            client.start();
            boolean connected = client.blockUntilConnected(timeout, TimeUnit.MILLISECONDS);
            if (!connected) {
                throw new IllegalStateException("zookeeper not connected");
            }
        } catch (Exception e) {
            throw new IllegalStateException(e.getMessage(), e);
        }
    }
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34

可以看出 是 boolean connected = client.blockUntilConnected(timeout, TimeUnit.MILLISECONDS); 这里 connected 返回了fasle导致的问题。
然后下端点， 发现timeout = 3000 ，也就说3秒内需要链接成功，否则就会超时。

推测由于vpn的方式连接的 zk服务器，很有可能是 3秒内没有得到服务端的正确响应，而导致了异常，然后抛出了异常。

基于上的推测，将该timeout的默认值3000改大一些后，然后就启动成功了。

解决

修改dubbo的配置，下面是springboot的项目，修改的方式。
从上面的报错信息是可以看出是 configcenter的连接超时。下面配置了三个配置的超时时间，可根据情况进行配置。

dubbo:
  config-center:
    timeout: 60000
  metadata-report:
    timeout: 60000
  registry:
    timeout: 60000
1
2
3
4
5
6
7

注意： 有可能版本的问题，上面3个属性，全部配置了，有可能项目启动报错， 请根据保持信息，适当调整删减属性后重试。