如何使用Terraform - GCP BigQuery在单个/多个数据集中创建多个表
如何使用Terraform - GCP BigQuery在单个/多个数据集中创建多个表
提问于 2022-07-14 17:02:03
目前，我能够在每个数据集下创建单个表，下面是我创建的代码；
main.tf
#############################################################
# BIGQUERY DATASET
#############################################################
resource "google_bigquery_dataset" "bq_dataset" {
for_each = var.bq_tables_schema
dataset_id = each.value["dataset_id"]
friendly_name = each.value["friendly_name"]
}
#############################################################
# BIGQUERY TABLE
#############################################################
resource "google_bigquery_table" "tables" {
for_each = var.bq_tables_schema
dataset_id = each.value["dataset_id"]
table_id = each.value["table_id"]
schema = each.value["schema"]
depends_on = [google_bigquery_dataset.bq_dataset]
}这是我使用的Terraform版本；
version.tf
terraform {
required_providers {
google = {
source = "hashicorp/google"
version = "~> 3.63.0"
}
}
required_version = ">= 0.13"
}下面是变量文件；
variable.tf
bq_tables_schema = {
"dataset1" = {
"dataset_id" = "bq_dataset_sentinel_test_1",
"friendly_name" = "bq_dataset_sentinel_test_1",
"table_id" = "bq_table_1",
"schema" = <<EOF
[
{
"name": "scan_time1",
"type": "STRING",
"mode": "NULLABLE",
"description": "scan time1"
},
{
"name": "scan_duration1",
"type": "STRING",
"mode": "NULLABLE",
"description": "scan duration1"
},
{
"name": "scm_url",
"type": "STRING",
"mode": "NULLABLE",
"description": "scm url"
}
]
EOF
},
"dataset2" = {
"dataset_id" = "bq_dataset_sentinel_test_2",
"friendly_name" = "bq_dataset_sentinel_test_2",
"table_id" = "bq_table_11",
"schema" = <<EOF
[
{
"name": "scan_time11",
"type": "STRING",
"mode": "NULLABLE",
"description": "scan time11"
},
{
"name": "scan_duration11",
"type": "STRING",
"mode": "NULLABLE",
"description": "scan duration11"
}
]
EOF
}
}一切看起来都很好，但是现在我试图在每个数据集中创建多个表，我怎么做呢？
下面是变量；我试图在每个/多个数据集下创建多个表:如何将该变量扁平化？
bq_tables_schema = {
"dataset1" = {
"dataset_id" = "bq_dataset_sentinel_test_1",
"friendly_name" = "bq_dataset_sentinel_test_1",
"table_schema" = {
"table_1" = {
"table_id" = "bq_table_1",
"schema" = <<EOF
[
{
"name": "scan_time1",
"type": "STRING",
"mode": "NULLABLE",
"description": "scan time1"
},
{
"name": "scan_duration1",
"type": "STRING",
"mode": "NULLABLE",
"description": "scan duration1"
},
{
"name": "scm_url",
"type": "STRING",
"mode": "NULLABLE",
"description": "scm url"
}
]
EOF
},
"table_2" = {
"table_id" = "bq_table_2",
"schema" = <<EOF
[
{
"name": "scan_time2",
"type": "STRING",
"mode": "NULLABLE",
"description": "scan time2"
},
{
"name": "scan_duration2",
"type": "STRING",
"mode": "NULLABLE",
"description": "scan duration2"
},
{
"name": "scm_url2",
"type": "STRING",
"mode": "NULLABLE",
"description": "scm url2"
}
]
EOF
}
}
},
"dataset2" = {
"dataset_id" = "bq_dataset_sentinel_test_2",
"friendly_name" = "bq_dataset_sentinel_test_2",
"table_schema" = {
"table_1" = {
"table_id" = "bq_table_11",
"schema" = <<EOF
[
{
"name": "scan_time11",
"type": "STRING",
"mode": "NULLABLE",
"description": "scan time11"
},
{
"name": "scan_duration11",
"type": "STRING",
"mode": "NULLABLE",
"description": "scan duration11"
}
]
EOF
},
"table_2" = {
"table_id" = "bq_table_22",
"schema" = <<EOF
[
{
"name": "scan_time22",
"type": "STRING",
"mode": "NULLABLE",
"description": "scan time22"
},
{
"name": "scan_duration22",
"type": "STRING",
"mode": "NULLABLE",
"description": "scan duration22"
}
]
EOF
}
}
}
}有人能帮我一下吗？
提前感谢！
回答 2
Stack Overflow用户
发布于 2022-07-17 04:42:03
应将数据集与表分开创建。
然后将生成的数据集用作每个表的变量。
您可以在这里找到更多信息：https://registry.terraform.io/providers/hashicorp/google/latest/docs/resources/bigquery_table
Stack Overflow用户
发布于 2022-07-25 14:15:03
我只是想出解决方案，用多个表创建多个数据集；
我在扁平的呃变量中添加了locals.tf，
local.tf
locals {
dataset_tables = flatten([
for dataset, dataset_info in var.bq_tables_schema : [
for table, table_data in dataset_info.table_schema : {
dataset_id = dataset_info.dataset_id
friendly_name = dataset_info.friendly_name
table_id = table_data.table_id
schema = table_data.schema
}
]
])
}这里是main.tf；
main.tf
#############################################################
# BIGQUERY DATASET
#############################################################
resource "google_bigquery_dataset" "bq_dataset" {
for_each = var.bq_tables_schema
dataset_id = each.value["dataset_id"]
friendly_name = each.value["friendly_name"]
}
#############################################################
# BIGQUERY TABLE
#############################################################
resource "google_bigquery_table" "tables" {
for_each = {
for dt in local.dataset_tables : "${dt.dataset_id}-${dt.table_id}" => dt
}
dataset_id = each.value.dataset_id
table_id = each.value.table_id
schema = each.value.schema
depends_on = [google_bigquery_dataset.bq_dataset]
}这是variables.tf文件
bq_tables_schema = {
"dataset1" = {
"dataset_id" = "bq_dataset_sentinel_test_1",
"friendly_name" = "bq_dataset_sentinel_test_1",
"table_schema" = {
"table_1" = {
"table_id" = "bq_table_1",
"schema" = <<EOF
[
{
"name": "scan_time1",
"type": "STRING",
"mode": "NULLABLE",
"description": "scan time1"
},
{
"name": "scan_duration1",
"type": "STRING",
"mode": "NULLABLE",
"description": "scan duration1"
},
{
"name": "scm_url",
"type": "STRING",
"mode": "NULLABLE",
"description": "scm url"
}
]
EOF
},
"table_2" = {
"table_id" = "bq_table_2",
"schema" = <<EOF
[
{
"name": "scan_time2",
"type": "STRING",
"mode": "NULLABLE",
"description": "scan time2"
},
{
"name": "scan_duration2",
"type": "STRING",
"mode": "NULLABLE",
"description": "scan duration2"
},
{
"name": "scm_url2",
"type": "STRING",
"mode": "NULLABLE",
"description": "scm url2"
}
]
EOF
}
}
},
"dataset2" = {
"dataset_id" = "bq_dataset_sentinel_test_2",
"friendly_name" = "bq_dataset_sentinel_test_2",
"table_schema" = {
"table_1" = {
"table_id" = "bq_table_11",
"schema" = <<EOF
[
{
"name": "scan_time11",
"type": "STRING",
"mode": "NULLABLE",
"description": "scan time11"
},
{
"name": "scan_duration11",
"type": "STRING",
"mode": "NULLABLE",
"description": "scan duration11"
}
]
EOF
},
"table_2" = {
"table_id" = "bq_table_22",
"schema" = <<EOF
[
{
"name": "scan_time22",
"type": "STRING",
"mode": "NULLABLE",
"description": "scan time22"
},
{
"name": "scan_duration22",
"type": "STRING",
"mode": "NULLABLE",
"description": "scan duration22"
}
]
EOF
}
}
}
}页面原文内容由Stack Overflow提供。腾讯云小微IT领域专用引擎提供翻译支持
原文链接：
https://stackoverflow.com/questions/72984205
复制相关文章
相似问题
腾讯云开发者
Copyright © 2013 - 2026 Tencent Cloud. All Rights Reserved. 腾讯云 版权所有
深圳市腾讯计算机系统有限公司 ICP备案/许可证号：粤B2-20090059 
粤公网安备44030502008569号
腾讯云计算（北京）有限责任公司 京ICP证150476号 | 京ICP备11018762号