使用Kubernetes dns add-on发现多个Hazelcast实例的问题

Question

我已经使用JHipster创建了一个整体式应用程序，并希望在GCP Kubernetes引擎集群上运行它的几个副本。

因为我想在副本(甚至是集群上的服务)之间共享数据，所以我决定使用内置的int Hazelcast缓存，但我不想使用推荐的尤里卡发现客户端，而是希望使用K8 dns-discovery附加组件。

所以我在我的pom.xml中添加了：

<dependency>
    <groupId>com.hazelcast</groupId>
    <artifactId>hazelcast-kubernetes</artifactId>
    <version>1.1.0</version>
</dependency>

并使用生成的CacheConfiguration.java文件来支持hazelcast-kubernetes配置(基于他们的github存储库)：

@Bean
public HazelcastInstance hazelcastInstance(JHipsterProperties jHipsterProperties) {
    log.debug("Configuring Hazelcast");
    HazelcastInstance hazelCastInstance = Hazelcast.getHazelcastInstanceByName("hazelcastK8");
    if (hazelCastInstance != null) {
        log.debug("Hazelcast already initialized");
        return hazelCastInstance;
    }
    Config config = new Config();
    config.setInstanceName("hazelcastK8");
    config.getNetworkConfig().setPort(5701);
    config.getNetworkConfig().setPortAutoIncrement(true);

    config.getNetworkConfig().getJoin().getMulticastConfig().setEnabled(false);
    config.getNetworkConfig().getJoin().getTcpIpConfig().setEnabled(false);

    if (env.acceptsProfiles(JHipsterConstants.SPRING_PROFILE_DEVELOPMENT)) {
        System.setProperty("hazelcast.local.localAddress", "127.0.0.1");
        config.getNetworkConfig().getJoin().getAwsConfig().setEnabled(false);
    } else { // In production we want to use hazelcast k8's dns discovery service
        log.info("Configuring hazelcast DNS discovery by K8");

        DiscoveryStrategyConfig strategyConfig =
            new DiscoveryStrategyConfig(
                new HazelcastKubernetesDiscoveryStrategyFactory()
            );

        // Default namespace
        strategyConfig.addProperty("service-dns", "hazelcastk8.default.svc.cluster.local");
        strategyConfig.addProperty("service-dns-timeout", 10);

        config.getNetworkConfig().getJoin().getDiscoveryConfig().addDiscoveryStrategyConfig(strategyConfig);
    }

    config.getMapConfigs().put("default", initializeDefaultMapConfig());
    config.setManagementCenterConfig(initializeDefaultManagementCenterConfig(jHipsterProperties));
    config.getMapConfigs().put("com.company.distributed.domain.*", initializeDomainMapConfig(jHipsterProperties));

    return Hazelcast.newHazelcastInstance(config);
}

为了检查数据网格是否如预期的那样工作，我编写了一个简单的调度任务，打印原子-long的当前值：

@Service
@Profile(JHipsterConstants.SPRING_PROFILE_PRODUCTION)
public class FetchHazelcastClusterSequence {

    private final HazelcastInstance hazelcastInstance;

    private static final Logger log = LoggerFactory.getLogger(FetchHazelcastClusterSequence.class);

    public FetchHazelcastClusterSequence(HazelcastInstance hazelcastInstance) {
        this.hazelcastInstance = hazelcastInstance;
    }

    @Timed
    @Scheduled(fixedDelay = 5000)
    public void start() {
        log.info("Fetching cluster sequence, the current value is: {}", this.hazelcastInstance.getAtomicLong("sequence").getAndIncrement());
    }
}

但当我启动应用程序时，默认配置为3个副本，创建了3个pod，但每个pod日志都显示为集群中的唯一成员，并因此打印相同的“序列”值。

我是否遗漏了一些配置？

Answer 1

@lior-ziv，您的配置缺少一个重要的配置参数：

1-使用SPRING_PROFILE_DEVELOPMENT配置文件时，您没有启用任何发现机制。您需要启用TCP-IP发现，否则所有成员都将启动单个节点集群。

2-当使用Kubernetes配置文件时，你有SPRING_PROFILE_PRODUCTION发现插件，但是你根本没有启用发现。这就是每次启动新实例时，它还会创建单个成员集群的原因。

正确的配置应如下所示：

//Move this like outside of if block since it'll be disabled for both cases.
config.getNetworkConfig().getJoin().getAwsConfig().setEnabled(false);

if (env.acceptsProfiles(JHipsterConstants.SPRING_PROFILE_DEVELOPMENT)) {
    config.getNetworkConfig().getJoin().getTcpIpConfig().setEnabled(true);
    config.getNetworkConfig().getJoin().getTcpIpConfig().addMember("127.0.0.1");`
} else {
    // In production we want to use hazelcast k8's dns discovery service
    System.setProperty("hazelcast.discovery.enabled", "true");
    log.info("Configuring hazelcast DNS discovery by K8");

    DiscoveryStrategyConfig strategyConfig =
        new DiscoveryStrategyConfig(
            new HazelcastKubernetesDiscoveryStrategyFactory()
        );

    // Default namespace
    strategyConfig.addProperty("service-dns", "hazelcastk8.default.svc.cluster.local");
    strategyConfig.addProperty("service-dns-timeout", 10);

    config.getNetworkConfig().getJoin().getDiscoveryConfig().addDiscoveryStrategyConfig(strategyConfig);
}