发布Spring引导中的大型数据的最佳方法

Question

我们有Elastic，它有员工任务的详细信息，我们每天使用spring引导应用程序向Kafka发布针对员工的任务。

弹性DB指数: employee_task

    {
    "employeeId":"E001",
    "taskName":"task1",
    "taskDesc":"task desc",
    "startDate":"2022-10-10 11:00:00",
    "endDate":"2022-10-10 16:00:00"
    }
    {
    "employeeId":"E001",
    "taskName":"task2",
    "taskDesc":"task desc",
    "startDate":"2022-10-10 16:00:00",
    "endDate":"2022-10-10 18:02:00"
    }
    {
    "employeeId":"E002",
    "taskName":"task3",
    "taskDesc":"task desc",
    "startDate":"2022-10-10 09:00:00",
    "endDate":"2022-10-10 18:00:00"
    }

弹簧启动代码：

    @Scheduled(cron = "${cron.task.expression}")
    public void scheduleTasks() {
                //Get District Employee Ids from index employee_task
                List<String> employees = taskService.getAllEmployeeIds();   
                //fetch tasks from index employee_task for each employee and publish to Kafka      
                employees.parallelStream().forEach(employeeId -> {
                    Map<String, Object> tasksList = taskService.getAllTasksByEmployeeId(employeeId);
                    kafkaTemplate.send(topicName, mapper.writeValueAsString(tasksList));
                });
    }

它将每天以以下格式向Kafka发布任务细节，

Message.1
{
"employeeId":"E001",
"taskList":[
    {
    "employeeId":"E001",
    "taskName":"task1",
    "taskDesc":"task desc",
    "startDate":"2022-10-10 11:00:00",
    "endDate":"2022-10-10 16:00:00"
    }
    {
    "employeeId":"E001",
    "taskName":"task2",
    "taskDesc":"task desc",
    "startDate":"2022-10-10 16:00:00",
    "endDate":"2022-10-10 18:02:00"
    }
]
}
Message.2
{
"employeeId":"E002",
"taskList":[
    {
    "employeeId":"E002",
    "taskName":"task3",
    "taskDesc":"task desc",
    "startDate":"2022-10-10 09:00:00",
    "endDate":"2022-10-10 18:00:00"
    }
]
}

直到现在，一切都很好，因为数据很低。但现在

Current No. of employees: 10,000
Average Task per Employee: 100

所以当cron运行时，它会查询10K的弹性DB。有人能建议怎样才能最好地处理这样的案件吗？

Answer 1

在DB世界中有句老话：“逐行等于慢而慢”。考虑到问题中提供的内容，我猜服务在某种循环中调用DB。需要编写一些方法来一次从DB中获取数据。