php mysql如何分割大表

数据库log表有几千万条数据,导致前台用户读取log信息时很慢,特别时做一些按月按日统计时,直接查询大表 会把网站搞崩溃。

这时候 需要把log表拆分为一个统计表,比如按日统计的表(log_day)。那么前台读取时 就直接读取这个log_day就行了。

因为数据量太大,所以直接用sql来insert in from select ..,如果用php先读取 再写入的话 内存也会奔溃。

 

可以用pdo的事务来写代码。如果有报错 就自动回滚。

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
<?php
if(php_sapi_name() != "cli")die;
$config= dirname(__FILE__)."/config.php";
$config = require($config);
$databaseConfig = $config['DB']['read'][0];
try {
$dbh = new PDO("mysql:host={$databaseConfig['host']};dbname={$databaseConfig['name']}", $databaseConfig['user'], $databaseConfig['passwd']);
echo "Connected\n";
} catch (Exception $e) {
die("Unable to connect: " . $e->getMessage());
}
try {
$dbh->setAttribute(PDO::ATTR_ERRMODE, PDO::ERRMODE_EXCEPTION);
$dbh->beginTransaction();

//fetch associative arrays
//$dbh->setAttribute(PDO::ATTR_DEFAULT_FETCH_MODE,PDO::FETCH_ASSOC);
//default chartset
$dbh->exec('SET NAMES "utf8"');
$sql = "
SET @@group_concat_max_len = 1024000;
truncate table log_day;
";
$dbh->exec($sql);
$sql = "INSERT INTO log_day (user_id,date,min_logid,max_logid,intro)
SELECT user_id,DATE_FORMAT(`datetime`,'%Y-%m-%d') as date,min(`id`) as min_logid,max(`id`) as max_logid,group_concat(`intro` separator '|||') as intro FROM log
group by date,user_id order by date;";
$dbh->exec($sql);
$sql = "UPDATE log_day as bld SET prev_balance = (SELECT prev_balance FROM log WHERE id=bld.`min_logid`),next_balance = (SELECT next_balance FROM log WHERE id=bld.`max_logid`)";
$dbh->exec($sql);
$dbh->commit();
//1,select all group by day

// $sth = $dbh->prepare($sql);
// $sth->execute();

// $sth = $dbh->prepare($sql);
// $sth->execute();

// $sth = $dbh->prepare($sql);
// $sth->execute();
//$result = $sth->fetchAll(PDO::FETCH_ASSOC);

//print_r($result);die;
$dbh = null;
} catch (PDOException $e) {
$dbh->rollBack();
echo "Failed: " . $e->getMessage();
}

 

如果要定时执行的话,就crontab -e 加入就行。

坚持原创技术分享,您的支持将鼓励我继续创作!