使用 Java 将 CSV 提取到 QuestDB 时出现“意外符号”

Question

我正在尝试获取 QuestDB 的示例数据集之一。整个文件可以在链接中看到，但第一行看起来像这样：

"symbol","side","price","amount","timestamp"
"BTC-USD","sell",25741.02,0.02188038,"2023-09-05T16:00:01.281719Z"
"BTC-USD","buy",25741.03,0.00184646,"2023-09-05T16:00:01.775613Z"
"BTC-USD","buy",25741.03,3.844E-5,"2023-09-05T16:00:02.722748Z"

我想使用 Java 和 REST API 提取数据。我有这段代码，但它给了我一个错误

import org.springframework.core.io.FileSystemResource;
import org.springframework.http.HttpEntity;
import org.springframework.http.HttpHeaders;
import org.springframework.http.MediaType;
import org.springframework.http.ResponseEntity;
import org.springframework.util.LinkedMultiValueMap;
import org.springframework.util.MultiValueMap;
import org.springframework.web.client.RestTemplate;

import java.io.File;
import java.util.Map;

public class Test {
    public static void main(String[] arg) {
        try {
            String url = "http://127.0.0.1:9000/imp?fmt=json&overwrite=true&forceHeader=true";
            String filePath = "/Users/admin/btc_trades.csv";

            RestTemplate restTemplate = new RestTemplate();
            HttpHeaders headers = new HttpHeaders();
            headers.setContentType(MediaType.MULTIPART_FORM_DATA);
            MultiValueMap<String, Object> body = new LinkedMultiValueMap<>();
            String schema = "[" +
                    "{\"name\":\"symbol\", \"type\": \"varchar\"}," +
                    "{\"name\":\"side\", \"type\": \"varchar\"}" +
                    "{\"name\":\"price\", \"type\": \"double\"}" +
                    "{\"name\":\"amount\", \"type\": \"double\"}" +
                    "{\"name\":\"timestamp\", \"type\": \"timestamp\"}" +
                    "]";
            body.add("schema", schema);
            body.add("data", new FileSystemResource(new File(filePath)));
            HttpEntity<MultiValueMap<String, Object>> requestEntity = new HttpEntity<>(body, headers);
            ResponseEntity<Map> response = restTemplate.postForEntity(url, requestEntity, Map.class);

            System.out.println("Response Status Code: " + response.getStatusCode());
            System.out.println("Response Headers:" + response.getHeaders());
            System.out.println("Response Body: " + response.getBody());
        } catch (Exception e) {
            System.out.println("Exception:" + e.getMessage());
            e.printStackTrace();
        }
    }
}

我得到以下输出：

Response Status Code: 200 OK
Response Headers:[Server:"questDB/1.0", Date:"Mon, 18 Nov 2024 05:53:23 GMT", Transfer-Encoding:"chunked", Content-Type:"application/json; charset=utf-8"]
Response Body: {status=Unexpected symbol}

任何人都可以看出上面的程序有什么问题或缺失。

Answer 1

这里有两个问题。第一个是架构的 JSON 字符串构造得不好，并且列定义末尾缺少尾随逗号。修复逗号后的第二个问题是，当我们在模式中传递时间戳时，我们需要添加时间戳的模式。如果我们将模式定义更改为此，它应该可以工作。

 String schema = "[" +
                    "{\"name\":\"symbol\", \"type\": \"varchar\"}," +
                    "{\"name\":\"side\", \"type\": \"varchar\"}," +
                    "{\"name\":\"price\", \"type\": \"double\"}," +
                    "{\"name\":\"amount\", \"type\": \"double\"}," +
                    "{\"name\":\"timestamp\", \"type\": \"timestamp\", \"pattern\": \"yyyy-MM-ddTHH:mm:ss.SSSUUUZ\"}" +
                    "]";

另一种方法是预先创建表（使用 REST API、pgwire 接口或仅从 Web 控制台交互），因此我们在导入时不必传递架构。手动定义表还允许配置重要的内容，例如符号容量或重复数据删除。