如何使用Python多线程处理MySQL连接

Question

我有一个主要的Python脚本连接到MySQL数据库并从中提取少量记录。根据返回的结果，它会启动尽可能多的线程（类实例），因为抓取了很多记录。每个线程应返回数据库并通过将一个状态标志设置为不同的状态来更新另一个表（“已启动进程”）。

为实现这一目标，我尝试：

1.）将数据库连接传递给所有线程2.）从每个线程打开一个新的数据库连接

但他们都没有工作。

我可以通过使用try / except在两种情况下都没有任何问题地运行我的更新，但MySQL表尚未更新，并且没有生成错误。我在两种情况下都使用了commit。

我的问题是如何在这种情况下处理MySQL连接？

根据前几条评论进行更新：

MAIN SCRIPT
-----------

#Connecting to DB
db = MySQLdb.connect(host = db_host,
                         db = db_db,
                         port = db_port,
                         user = db_user,
                         passwd = db_password,
                         charset='utf8')

# Initiating database cursor
cur = db.cursor()

# Fetching records for which I need to initiate a class instance

cur.execute('SELECT ...')

for row in cur.fetchall() :
    # Initiating new instance, appending it to a list and
    # starting all of them 



CLASS WHICH IS INSTANTIATED
---------------------------

# Connecting to DB again. I also tried to pass connection
# which has been opened in the main script but it did not
# work either.

db = MySQLdb.connect(host = db_host,
                         db = db_db,
                         port = db_port,
                         user = db_user,
                         passwd = db_password,
                         charset='utf8')

# Initiating database cursor
cur_class = db.cursor()
cur.execute('UPDATE ...')
db.commit()

Answer 1

下面是一个在Python中使用多线程处理mysql的例子，我不知道你的表和数据，所以，只需更改代码可能会有所帮助：

import threading
import time
import MySQLdb

Num_Of_threads = 5

class myThread(threading.Thread):

    def __init__(self, conn, cur, data_to_deal):
        threading.Thread.__init__(self)
        self.threadID = threadID
        self.conn = conn
        self.cur = cur
        self.data_to_deal

    def run(self):

        # add your sql 
        sql = 'insert into table id values ({0});'
        for i in self.data_to_deal:
            self.cur.execute(sql.format(i))
            self.conn.commit()

threads = []
data_list = [1,2,3,4,5]

for i in range(Num_Of_threads):
    conn = MySQLdb.connect(host='localhost',user='root',passwd='',db='')
    cur = conn.cursor()
    new_thread = myThread(conn, cur, data_list[i])

for th in threads:
    th.start()

for t in threads:
    t.join()

Answer 2

似乎我的代码没有问题，但我的MySQL版本。我正在使用MySQL标准社区版并基于找到here的官方文档：

线程池插件是一个商业功能。它不包含在MySQL社区发行版中。

我即将升级到MariaDB来解决这个问题。

Answer 3

看起来mysql 5.7确实支持多线程。

正如您之前尝试的那样 - 绝对确保在def worker（）中传递连接。全局定义连接是我的错误

这是通过5个线程打印10条记录的示例代码，共5次

import MySQLdb
import threading


def write_good_proxies():    
    local_db = MySQLdb.connect("localhost","username","PassW","DB", port=3306 )
    local_cursor = local_db.cursor (MySQLdb.cursors.DictCursor)
    sql_select = 'select http from zproxies where update_time is null order by rand() limit 10'
    local_cursor.execute(sql_select)
    records = local_cursor.fetchall()
    id_list = [f['http'] for f in records]
    print id_list
def worker():
    x=0
    while x< 5:
        x = x+1
        write_good_proxies()

threads = []


for i in range(5):
    print i
    t = threading.Thread(target=worker)
    threads.append(t)
    t.start()