SISS 包未将所有记录从 Api 提取到数据库

Question

我正在尝试将数据从 Json Rest API 移动到我的数据库。我正在使用数据流任务，其中脚本任务作为源，OLE DB 作为目标。

样本数据

{
    "Id": 1,
    "Name": "Project A",
    "owner": "User A",
    "status": "Active",
    "listedItems": [
        {"column1": "100", "column2": "1000", "column3": "Type A"},
        {"column1": "200", "column2": "2000", "column3": "Type B"},
        {"column1": "300", "column2": "3000", "column3": "Type C"}
    ]
}

我只想将column1、column2和column3数据移动到数据库中。

    using System;
    using System.Data;
    using Microsoft.SqlServer.Dts.Pipeline.Wrapper;
    using Microsoft.SqlServer.Dts.Runtime.Wrapper;
    using System.Collections.Generic;
    using System.IO;
    using System.Net;
    using static System.Windows.Forms.VisualStyles.VisualStyleElement.Tab;
    using System.Text.RegularExpressions;

    public class ScriptMain : UserComponent
     {
         public override void CreateNewOutputRows()    
         {
             // Specify the security protocol
             System.Net.ServicePointManager.SecurityProtocol = SecurityProtocolType.Tls12;
    
             // Define the API URL
             string apiUrl = "URL"; // Update with your actual API URL
             string apiKey = "apiKey";
    
             try
             {
                 HttpWebRequest request = (HttpWebRequest)WebRequest.Create(apiUrl);
                 request.Method = "GET";
                 request.Headers["Authorization"] = "Bearer " + apiKey;
    
                 using (HttpWebResponse response = (HttpWebResponse)request.GetResponse())
                 {
                     using (StreamReader reader = new StreamReader(response.GetResponseStream()))
                     {
                         string jsonResponse = reader.ReadToEnd();
                         ParseInventoryItems(jsonResponse);
                     }
                 }
             }
             catch (Exception ex)
             {
                 System.Diagnostics.Debug.WriteLine("Error: " + ex.Message);
             }
         }
    
         private void ParseInventoryItems(string jsonResponse)
         {
             try
             {
                 // Extract inventory items as before
                 string pattern = "\"listedItems\":\\[(.*?)\\]";
                 Match match = Regex.Match(jsonResponse, pattern);
    
                 if (match.Success)
                 {
                     string listedItemsContent = match.Groups[1].Value;
                     var items = listedItemsContent.Split(new string[] { "},{" }, StringSplitOptions.None);
    
                     int rowCount = 0;  // Count of rows added
                     foreach (var item in items)
                     {
                         string cleanedItem = item.Replace("{", "").Replace("}", "").Trim();
                         var keyValuePairs = cleanedItem.Split(',');
    
                         Output0Buffer.AddRow();  // Add a new row
    
                         // Initialize values
                         Output0Buffer.column1 = string.Empty;
                         Output0Buffer.column2 = string.Empty;
                         Output0Buffer.column3 = string.Empty;
    
                         // Parse key-value pairs
                         foreach (var pair in keyValuePairs)
                         {
                             var keyValue = pair.Split(':');
                             if (keyValue.Length != 2) continue;
    
                             string key = keyValue[0].Trim().Trim('"');
                             string value = keyValue[1].Trim().Trim('"');
    
                             // Set output buffer values
                             if (key.Equals("column1", StringComparison.OrdinalIgnoreCase))
                             {
                                 Output0Buffer.column1 = value;
                             }
                             else if (key.Equals("column2", StringComparison.OrdinalIgnoreCase))
                             {
                                 Output0Buffer.column2 = value;
                             }
                             else if (key.Equals("tcolum3", StringComparison.OrdinalIgnoreCase))
                             {
                                 Output0Buffer.column3 = value;
                             }
                         }
    
                         rowCount++;  // Increment row count
                                 // Log the current row information
                         
                     }
    
                     // Final log of total rows processed
                     System.Diagnostics.Debug.WriteLine($"Total Rows Processed: {rowCount}");
                 }
                 else
                 {
                     System.Diagnostics.Debug.WriteLine("No listedItems found in JSON response.");
                 }
             }
             catch (Exception ex)
             {
                 System.Diagnostics.Debug.WriteLine("Error parsing inventory items: " + ex.Message);
             }
         }
     }

上面的代码仅提取列表中的第一项（“column1”：“100”，“column2”：“1000”，“column3”：“Type A”）如何获取该列表中的所有项目？

Answer 1

我认为您可能采取了错误的方式。假设您使用的是 SQL Server 2019+，您可以在数据库中使用本机 JSON 处理。您甚至不需要这里的数据流任务。

首先，确保将 API 结果保存到字符串变量中。然后您可以使用以下 SQL 语句来解析出您要查找的值。在 SSIS 中，您可以调整以下内容以在

Execute SQL Task

中使用。

以您的 JSON 响应为例，我想出了这个：

DECLARE @JSON NVARCHAR(MAX) = N'{
    "Id": 1,
    "Name": "Project A",
    "owner": "User A",
    "status": "Active",
    "listedItems": [
        {"column1": "100", "column2": "1000", "column3": "Type A"},
        {"column1": "200", "column2": "2000", "column3": "Type B"},
        {"column1": "300", "column2": "3000", "column3": "Type C"}
    ]
}'

SELECT b.* 
FROM OPENJSON(@JSON)
    WITH 
        (listedItems NVARCHAR(MAX) AS JSON) AS a
CROSS APPLY OPENJSON(a.listedItems) 
    WITH
        (
            column1 INT N'$.column1'
            ,column2 INT N'$.column2'
            ,column3 NVARCHAR(MAX) N'$.column3'
        ) AS b

如果您将其改编为在

Execute SQL Task

中使用，则可以使用 SELECT 语句作为表的一部分直接插入到目标表中。