Laravel 504 网关超时

Question

我在 Laravel 平台中有一个脚本可以处理数千条记录，并且存在超时问题。我想避免更新服务器上的设置，例如 php.ini 文件。

有没有办法将所有记录分成小块，并在进一步处理中使用它们，直到使用所有记录。

我有数千条记录正在从另一台服务器获取价格。

那么，

1-10 条记录工作正常。

1-100 条记录就可以了。

超过 500 条记录脚本卡住并给出如下错误。

405 不允许 nginx 服务器

我的路线就像：- example.com/updateAllKeywords

下面的代码是获取关键字以从 eBay API 获取关键字的价格

        $keywordCollection = Keywords::all();

        Keywords::chunk(100, function ($keywordCollection) use($request,$keywordCollection,$defaultStartDate,$ignoredItems,$response) {
            foreach ($keywordCollection->toArray() as $keyword_result) {
                $response = $this->apirequest($request, $keyword_result['id'], $keyword_result, $defaultStartDate, $ignoredItems);
            }
        });

这是api请求函数：

 /**
 * Call Finding API for Active List items 
 *
 * @return API Response
 */
public function apirequest(Request $request, $id, $keyword_result, $defaultStartDate, $ignoredItems) {

    try {

        /* Request */
        $request = new Types\FindItemsAdvancedRequest();

        /* 1) default 100 items being displayed without pagination */
        $request->paginationInput = new Types\PaginationInput();
        $request->paginationInput->entriesPerPage = Config::get('constants.paginationoptions.entries_per_page');

        /* 2) ItemFilter for three conditions NEW ,NEWOTHER,USED */
        $itemFilterForConditions = new Types\ItemFilter();
        $itemFilterForConditions->name = 'Condition';
        $itemFilterForConditions->value = ['1000', '1500', '3000'];

        /* 3) ItemFilter for hide duplicate items */
        $itemFilterduplicate = new Types\ItemFilter();
        $itemFilterduplicate->name = 'HideDuplicateItems';
        $itemFilterduplicate->value[] = 'true';

        /* 4) pull items before given date / days */
        $itemFilterStartDateFrom = new Types\ItemFilter();
        $itemFilterStartDateFrom->name = 'StartTimeFrom';
        $itemFilterStartDateFrom->value[] = date('Y-m-d\TH:i:s\Z', mktime(0, 0, 0, date('m'), date('d') - $defaultStartDate['defaultStartDate'], date('y')));

        /* Exclude Keywords/MPN by set Priced Percentage */
        if (!empty($keyword_result['price']) && ($defaultStartDate['exclusionPercentage'] > 0)) {
            $percentageAmount = $keyword_result['price'] * ($defaultStartDate['exclusionPercentage'] / 100);
            $percentageAmount = floor($percentageAmount).substr($percentageAmount-floor($percentageAmount),1,2+1);
            $itemFilterPriced = new Types\ItemFilter();
            $itemFilterPriced->name = 'MinPrice';
            $itemFilterPriced->value[] = $percentageAmount;
        }

        $itemFilter = (isset($itemFilterPriced)) ? [$itemFilterForConditions, $itemFilterduplicate, $itemFilterStartDateFrom, $itemFilterPriced] : [$itemFilterForConditions, $itemFilterduplicate, $itemFilterStartDateFrom];

        $request->itemFilter = $itemFilter;

        /* 5) Request KEYWORD/MPN to filter Items Active Lists */
        $searchKeyword = (!empty($keyword_result['exclusions'])) ? $keyword_result['keyword'] . " -" . "(" . trim($keyword_result['exclusions']) . ")" : $keyword_result['keyword'];
        $request->keywords = $searchKeyword;

        /* 6) Request For sorting Items by Lowest To Highest Itemprice Price + Shipping Price */
        $request->sortOrder = 'PricePlusShippingLowest';

        /* 7) Request For Keyword Search Including the Item Descriptions,Title and Subtitle */
        $request->descriptionSearch = ($defaultStartDate['advancedExclusions'] == 1) ? true : false;

        $service = $this->index();

        /* Response returned by $request */
        $response = $this->apiresponse($request, $service, $id, $keyword_result['keyword'], $ignoredItems);

        return $response;

    } catch (\Exception $e) {
        echo json_encode(array('ack' => 'Failure', 'msg' => $e->getMessage()));
        exit;
    }
}


public function apiresponse($request, $service, $id, $keyword, $ignoredItems) {

            try {

                $response = $service->findItemsAdvanced($request);

                /* fetch all ignored Items */

                $newItems = array();
                $newOtherItems = array();
                $usedItems = array();
                $errors = "";

                if ($response->ack == "Success") {
                    $results = $response->searchResult->item;

                    ob_start();

                    foreach ($results as $item) {

                        if ($item->listingInfo->buyItNowAvailable == '1') {
                            $itemPrice = $item->listingInfo->buyItNowPrice->value;
                        } elseif (($item->listingInfo->listingType == 'Auction') && ($item->sellingStatus->currentPrice->value == 0)) {
                            $itemResponse = $this->getItemDetails($item->itemId);
                            $itemPrice = isset($itemResponse) ? $itemResponse : 0;
                        } else {
                            $itemPrice = $item->sellingStatus->currentPrice->value;
                        }

                        /* check for ignored items */
                        if (!in_array($item->itemId, $ignoredItems)) {

                            /* check for NEW Condition (conditionId = 1000) */
                            if ($item->condition->conditionId == '1000') {
                                $newItems[] = array(
                                    'keywordId' => $id,
                                    'conditionId' => (string) $item->condition->conditionId,
                                    'itemId' => $item->itemId,
                                    'title' => $item->title,
                                    'price' => $itemPrice,
                                    'itemURL' => $item->viewItemURL,
                                    'galleryURL' => isset($item->galleryURL) ? $item->galleryURL : '',
                                );
                            }
                            /* check for NEW OTHER Condition (conditionId = 1500) */ elseif ($item->condition->conditionId == '1500') {
                                $newOtherItems[] = array(
                                    'keywordId' => $id,
                                    'conditionId' => (string) $item->condition->conditionId,
                                    'itemId' => $item->itemId,
                                    'title' => $item->title,
                                    'price' => $itemPrice,
                                    'itemURL' => $item->viewItemURL,
                                    'galleryURL' => isset($item->galleryURL) ? $item->galleryURL : '',
                                );
                            }
                            /* check for USED Condition (conditionId = 3000) */ elseif ($item->condition->conditionId == '3000') {
                                $usedItems[] = array(
                                    'keywordId' => $id,
                                    'conditionId' => (string) $item->condition->conditionId,
                                    'itemId' => $item->itemId,
                                    'title' => $item->title,
                                    'price' => $itemPrice,
                                    'itemURL' => $item->viewItemURL,
                                    'galleryURL' => isset($item->galleryURL) ? $item->galleryURL : '',
                                );
                            }
                        }

                        ob_flush();
                    }

                    $newItems = array_slice($newItems, 0, 20);
                    $newOtherItems = array_slice($newOtherItems, 0, 20);
                    $usedItems = array_slice($usedItems, 0, 20);

                    $insertItems = array_merge($newItems, $newOtherItems, $usedItems);

                    /* clean up items */
                    Items::where('keywordId', $id)->delete();

                    /* save pulled items */
                    $items = Items::insert($insertItems);
                } elseif ($response->ack == "Failure") {
                    foreach ($response->errorMessage->error as $error) {
                        $errors .= "%s: %s\n\n" . $error->severity === Enums\ErrorSeverity::C_ERROR ? '<b>Error</b>&nbsp;' : '<b>Warning</b>&nbsp;' . $error->message;
                    }
                }

                /* save API log fot the keyword */
                $this->setapilog($request, $response, $keyword);

                if (!empty($errors)) {
                    echo json_encode(array('ack' => 'Failure', 'msg' => $errors));
                    exit;
                } else {
                    return true;
                }
            } catch (\Exception $e) {
                echo json_encode(array('ack' => 'Failure', 'msg' => $e->getMessage()));
                exit;
            }
        }

任何有关这方面的信息将不胜感激。谢谢！

Answer 1

创建此任务Laravel 队列 将您的函数放入句柄方法中。

$response = $this->apiRequest(...);

从您的控制器调度您的工作

foreach ($keywordCollection as $key => $keywordResult) {
   ProcessJob::dispatch($keywordResult)->delay($key * 2); 
}

通过这种方式，您可以实现指数退避，这样随着数组的增加，您会增加发出下一个 api 请求的等待时间。

Answer 2

@kinjal Jethva

问题：

您正在 foreach 循环中使用数据库查询。您正在将 apirequest() 方法调用到 foreach 循环中。

当你的 $keywords 数组有 500 条记录时。

你正在呼唤

    $keyword_result => 500 times. 

    $defaultStartDate = 500 times

您总共在一个 http 请求中调用了 1000 个数据库查询。

解决方案：

在 apirequest() 函数中调用 $defaultStartDate 是完全没有用的。 相反，在外部调用它并将值作为参数传递。所以它会调用一次。

$keyword_result 可以用更好的方式调用。

$defaultStartDate = Settings::select('defaultStartDate', 
'exclusionPercentage', 'advancedExclusions')->where('currentEnv', 1)->get()-
>first()->toArray();

$keywordValues = array_keys($keywords);

$keywordCollection = Keywords::whereIn('id', $keywordValues)-
>select('keyword', 'exclusions', 'price')->first()->toArray();

foreach ($keywordCollection as $keyword_result) {

$response = $this->apirequest($request, $keyword_result->id, 
$keyword_result, $defaultStartDate);

}

/**
  * Call Finding API for Active List items
  *
  * @return API Response
  */
function apirequest(Request $request, $id, $keyword_result, $defaultStartDate)
{
 // your code here.
}

这只是您可以使用的代码结构。