Databricks API - 将文件从 DevOps 存储库导入到 databricks 工作区

Question

我们的管道逻辑包括

Add-DatabricksDBFSFile

函数，用于将

.sh

文件从 DevOps 存储库移动到 databricks 中的 DBFS，以便以后创建集群（下面的代码）

Add-DatabricksDBFSFile -BearerToken $ADB_Token -Region $region `
-LocalRootFolder $initFolder -FilePattern "pyodbc.sh"  -TargetLocation '/init' -Verbose

New-DatabricksCluster -BearerToken $ADB_Token -Region $region -ClusterName $cname -SparkVersion $csparkV `
-NodeType $cnodeT -MinNumberOfWorkers $cminWorker -MaxNumberOfWorkers $cmaxWorker -AutoTerminationMinutes $cterm `
-InitScripts "dbfs:/init/pyodbc.sh" -UniqueNames -Update

不，不再支持此功能，我需要 从 DBFS 上的初始化脚本迁移。

我的第一次尝试是使用以下代码将文件从 devops 存储库复制到 databricks 工作区中已存在的文件夹

Init

。

$initFolder = $PSScriptRoot + '\InitScripts'
$localFilePath = Join-Path -Path $initFolder -ChildPath 'pyodbc.sh'
$databricksInstance = "https://adb-88888888888888.88.azuredatabricks.net"
$workspacePath = '/Workspace/Init/pyodbc.sh'


$fileContent = [System.IO.File]::ReadAllBytes($localFilePath)
$fileBase64 = [Convert]::ToBase64String($fileContent)
$apiUrl = "$databricksInstance/api/2.0/workspace/import"


$body = @{
    path = $workspacePath
    content = $fileBase64
    format = "SOURCE"  # The format to specify that this is a plain text/source file
    overwrite = $false  # Set to true to overwrite any existing file at the destination path
} | ConvertTo-Json


$response = Invoke-RestMethod -Uri $apiUrl -Method POST -Headers @{Authorization = $InternalBearerToken } -Body $body -ContentType "application/json"
Write-Output "Init script writing"
$response
Write-Output "Init script successfully written"

但我收到此错误

调用RestMethod： {"error_code":"INVALID_PARAMETER_VALUE","message":"zip 文件可能 无效或可能是不受支持的版本。”}

Answer 1

您好，尝试将语言参数添加到 http 调用中

$body = @{
    path = $workspacePath
    content = $fileBase64
    language = "PYTHON"
    format = "SOURCE"  # The format to specify that this is a plain text/source file
    overwrite = $false  # Set to true to overwrite any existing file at the destination path
} | ConvertTo-Json