How to Import Large Data Batches

Sometimes a number of entities must be imported from a portal via REST. When a number of entities is significant, it's not optimal to directly set up of a filter and individually pass start = start+50 to each next query. When using start >= 0 each time, a query for counting the number of elements, satisfying the filter, is executed as well. It results in an operational slowdown when a large number of elements and complex filtering are used.

That's why you can just pass start = -1 in case when you don't need a specific number of elements (for example, you just need 10 last records) or when you are importing all records as per filter.

This parameter disables execution of an element counting query, significantly speeding up the result's retrieval.

To execute the import, you must sort records by ID and add a filter condition for ID > value of last element. And increase its value with each next step. The value of the last element must be taken from the last value of retrieved result.

Condition to stop the import is an empty response, or when the result contains less than 50 elements.

Below is an example of code and a comparison with the classic approach in terms of execution time. Respectively, when having 2 387 743 leads, with the same permissions and filter, execution time has decreased from 49.9 seconds to 0,097 seconds.

Example

$tokenID = 'XXXXXXXXXXXXXXXXXXXXX';
$host = 'XXXX.bitrix24.com';
$user = 1;

/**
 * Start from null or from a specific previous step
 */
$leadID = 0;
$finish = false;

while (!$finish)
{
   /**
    * Execute until all data is retrieved and do not forget about delay between hits.
    * Or each time select only 50, starting from the element on which the last iteration had stopped.
   */

   $http = new \Bitrix\Main\Web\HttpClient();

   $http->setTimeout(5);
   $http->setStreamTimeout(50);

   $json = $http->post(
      'https://'.$host.'/rest/'.$user.'/'.$tokenID.'/crm.lead.list/',
      [
         'order' => ['ID' => 'ASC'],
         'filter' => ['>ID' => $leadID],
         'select' => ['ID', 'TITLE', 'DATE_CREATE'],
         'start' => -1
      ]
   );

   $result = \Bitrix\Main\Web\Json::decode($json);
   if (count($result['result']) > 0)
   {
      foreach ($result['result'] as $lead)
      {
         $leadID = $lead['ID'];
      }
      // Do something
   }
   else
   {
      $finish = true;
   }
}

/*
//Executed REST query results with the count used.
   Array
   (
      [result] => Array
      (
         [0] => Array()
         [1] => Array()
         .....
         [49] => Array()

      [next] => 50
      [total] => 2387743
      [time] => Array
      (
         [start] => 1581607213.4833
         [finish] => 1581607263.3997
         [duration] => 49.916450023651
         [processing] => 49.899916887283
         [date_start] => 2020-02-13T18:20:13+03:00
         [date_finish] => 2020-02-13T18:21:03+03:00
      )
)

//Executed REST query results without using the count.

   Array
   (
      [result] => Array
      (
         [0] => Array()
         [1] => Array()
         .....
         [1] => Array()

      [total] => 0
      [time] => Array
      (
         [start] => 1581609136.3857
         [finish] => 1581609136.4835
         [duration] => 0.097883939743042
         [processing] => 0.068500995635986
         [date_start] => 2020-02-13T18:52:16+03:00
         [date_finish] => 2020-02-13T18:52:16+03:00
      )
)
*/