Insertion Efficiency Of A Large Amount Of Data With Sql

I have a program that I use to read CSV file and insert the data into a database. I am having trouble with it because it needs to able to insert big records ( up to 10,000 rows ) o

Solution 1:

So I ended up using the fputcsv to put the data I changed into a new CSV file, then I used the LOAD DATA INFILE command to put the data from the new csv file into the table. This changed it from timing out at 120 secs for 1000 entries, to taking about 10 seconds to do 10,000 entries. Thank you to everyone that replied.

Solution 2:

I have this crazy idea: Could you run multiple parallels scripts, each one takes care of a bunch of rows from your CSV.

Some thing like this:

<?php// this tells linux to run the import.php in background,// and releases your caller script.//// do this several times, and you could increase the overal time$cmd = "nohup php import.php [start] [end] & &>/dev/null";
exec($cmd);

Also, have you tried to increase these limit of 50 bulk inserts to 100 or 500 for example?