批量执行select语句

Question

我可以通过

psycopg2.extras.execute_values(cur, sql, argslist, template=None, page_size=100, fetch=False)

批量执行更新、插入和删除 (https://www.psycopg.org/docs/extras.html)

但是，我不知道如何将该方法用于 select 语句。 我需要以

SELECT * FROM X IN (id1,id2,...id100000)

的形式拆分一个大型选择查询，并且想知道是否可以使用该库而不是为此编写自定义代码。

您能否提供一个示例或更好的方法，将语句分成每个查询大约 1000 个 id 的批次？

Answer 1

我用过的一种解决方案。

从这里开始：

如何将列表分割成大小相等的块？

def chunks(lst, n):
    """Yield successive n-sized chunks from lst."""
    for i in range(0, len(lst), n):
        yield lst[i:i + n]

然后：

create table chunk_test(id integer);
insert into chunk_test select generate_series(1, 100);

然后：

import psycopg2

con = psycopg2.connect("dbname=test user=postgres port=5452")
cur = con.cursor()

select_qry = 'select id from chunk_test where id = any(%s)'
id_list = [id for id in range(1, 101)]
for chunk in chunks(id_list, 10):
    cur.execute(select_qry, [chunk])
    print(cur.fetchall())

返回：

[(1,), (2,), (3,), (4,), (5,), (6,), (7,), (8,), (9,), (10,)]
[(11,), (12,), (13,), (14,), (15,), (16,), (17,), (18,), (19,), (20,)]
[(21,), (22,), (23,), (24,), (25,), (26,), (27,), (28,), (29,), (30,)]
[(31,), (32,), (33,), (34,), (35,), (36,), (37,), (38,), (39,), (40,)]
[(41,), (42,), (43,), (44,), (45,), (46,), (47,), (48,), (49,), (50,)]
[(51,), (52,), (53,), (54,), (55,), (56,), (57,), (58,), (59,), (60,)]
[(61,), (62,), (63,), (64,), (65,), (66,), (67,), (68,), (69,), (70,)]
[(71,), (72,), (73,), (74,), (75,), (76,), (77,), (78,), (79,), (80,)]
[(81,), (82,), (83,), (84,), (85,), (86,), (87,), (88,), (89,), (90,)]
[(91,), (92,), (93,), (94,), (95,), (96,), (97,), (98,), (99,), (100,)]