Azure 数据工厂将日期字符串转换为 unix 时间戳

Question

我有一个 ADF，我想用它在 CosmosDB 容器之间复制文档。在此过程中，我希望有一个带有硬编码值的新列“类型”。效果很好。

但是，源文档上有 2 个字段（发送/发送）是 ISO8601 日期，其中 UTC 偏移量 Z 定义为字符串。即

2022-11-04T22:59:59Z

转换为

1667602799

。

我还没有定义任何此类日期时间转换。我的源数据集仅使用

select * from c

作为查询。这些文档没有通用的模式，尽管大多数都有“to/from”字段。

当我查看预览数据，甚至下载 Excel 预览时，日期都会呈现为 ISO8601 格式。仅在接收 Cosmos 容器中，它们才会被格式化为 unix 时间戳。

这是我的规格：

{
    "name": "transform qa",
    "properties": {
        "type": "MappingDataFlow",
        "typeProperties": {
            "sources": [
                {
                    "dataset": {
                        "referenceName": "src_qa",
                        "type": "DatasetReference"
                    },
                    "name": "source1"
                }
            ],
            "sinks": [
                {
                    "dataset": {
                        "referenceName": "dst_qa",
                        "type": "DatasetReference"
                    },
                    "name": "sink1"
                }
            ],
            "transformations": [
                {
                    "name": "derivedColumn1"
                }
            ],
            "scriptLines": [
                "source(allowSchemaDrift: true,",
                "     validateSchema: false,",
                "     query: 'select * from c',",
                "     format: 'documentQuery',",
                "     systemColumns: false) ~> source1",
                "source1 derive(type = \"myType\") ~> derivedColumn1",
                "derivedColumn1 sink(allowSchemaDrift: true,",
                "     validateSchema: false,",
                "     deletable:false,",
                "     insertable:true,",
                "     updateable:true,",
                "     upsertable:true,",
                "     format: 'document',",
                "     partitionKey: ['/id'],",
                "     skipDuplicateMapInputs: true,",
                "     skipDuplicateMapOutputs: true) ~> sink1"
            ]
        }
    }
}

如何确保我的字段按照 src 中的定义进行复制而不是转换？

Answer 1

但是，源文档上有 2 个字段（发送/发送）是 ISO8601 日期，其中 UTC 偏移量 Z 定义为字符串。即

2022-11-04T22:59:59Z

转换为

1667602799

。

要将日期存储在源容器中，在源选项中提供查询

select * from c

后，您需要单击

Import projection

，因为您需要将源投影中的数据类型从

datetime

更改为

string

可以在下面看到。

下面是存储在我的源容器中的示例数据：

{
    "id": "1",
    "name": "John Doe",
    "from": "2022-01-01T12:00:00Z",
    "to": "2022-01-02T12:00:00Z",
    "description": "Sample document 1"
}

{
    "id": "2",
    "name": "Jane Smith",
    "from": "2022-02-01T08:00:00Z",
    "to": "2022-02-02T08:00:00Z",
    "description": "Sample document 2"
}

{
    "id": "3",
    "name": "Alice Johnson",
    "from": "2022-03-01T18:00:00Z",
    "to": "2022-03-02T18:00:00Z",
    "description": "Sample document 3"
}

将数据类型更改为

string

后，下面是通过添加新列

type

存储在另一个容器中的数据。

{
    "name": "John Doe",
    "description": "Sample document 1",
    "to": "2022-01-02T12:00:00Z",
    "id": "1",
    "from": "2022-01-01T12:00:00Z",
    "type": "myType3"
}

{
    "name": "Jane Smith",
    "description": "Sample document 2",
    "to": "2022-02-02T08:00:00Z",
    "id": "2",
    "from": "2022-02-01T08:00:00Z",
    "type": "myType3"
}

{
    "name": "Alice Johnson",
    "description": "Sample document 3",
    "to": "2022-03-02T18:00:00Z",
    "id": "3",
    "from": "2022-03-01T18:00:00Z",
    "type": "myType3"
}