3 # The author disclaims copyright to this source code. In place of
4 # a legal notice, here is a blessing:
6 # May you do good and not evil.
7 # May you find forgiveness for yourself and forgive others.
8 # May you share freely, never taking more than you give.
10 #***********************************************************************
12 # This file contains tests to ensure that the library handles malloc() failures
13 # correctly. The emphasis of these tests are the _prepare(), _step() and
16 # $Id: malloc3.test,v 1.24 2008/10/14 15:54:08 drh Exp $
18 set testdir [file dirname $argv0]
19 source $testdir/tester.tcl
20 source $testdir/malloc_common.tcl
22 # Only run these tests if memory debugging is turned on.
25 puts "Skipping malloc3 tests: not compiled with -DSQLITE_MEMDEBUG..."
30 # Do not run these tests if F2FS batch writes are supported. In this case,
31 # it is possible for a single DML statement in an implicit transaction
32 # to fail with SQLITE_NOMEM, but for the transaction to still end up
33 # committed to disk. Which confuses the tests in this module.
35 if {[atomic_batch_write test.db]} {
36 puts "Skipping malloc3 tests: atomic-batch support"
42 # Do not run these tests with an in-memory journal.
44 # In the pager layer, if an IO or OOM error occurs during a ROLLBACK, or
45 # when flushing a page to disk due to cache-stress, the pager enters an
46 # "error state". The only way out of the error state is to unlock the
47 # database file and end the transaction, leaving whatever journal and
48 # database files happen to be on disk in place. The next time the current
49 # (or any other) connection opens a read transaction, hot-journal rollback
50 # is performed if necessary.
52 # Of course, this doesn't work with an in-memory journal.
54 if {[permutation]=="inmemory_journal"} {
59 #--------------------------------------------------------------------------
60 # NOTES ON RECOVERING FROM A MALLOC FAILURE
62 # The tests in this file test the behaviours described in the following
63 # paragraphs. These tests test the behaviour of the system when malloc() fails
64 # inside of a call to _prepare(), _step(), _finalize() or _reset(). The
65 # handling of malloc() failures within ancillary procedures is tested
70 # Executing a statement is done in three stages (prepare, step and finalize). A
71 # malloc() failure may occur within any stage. If a memory allocation fails
72 # during statement preparation, no statement handle is returned. From the users
73 # point of view the system state is as if _prepare() had never been called.
75 # If the memory allocation fails during the _step() or _finalize() calls, then
76 # the database may be left in one of two states (after finalize() has been
79 # * As if the neither _step() nor _finalize() had ever been called on
80 # the statement handle (i.e. any changes made by the statement are
82 # * The current transaction may be rolled back. In this case a hot-journal
83 # may or may not actually be present in the filesystem.
85 # The caller can tell the difference between these two scenarios by invoking
89 # Handling of sqlite3_reset():
91 # If a malloc() fails while executing an sqlite3_reset() call, this is handled
92 # in the same way as a failure within _finalize(). The statement handle
93 # is not deleted and must be passed to _finalize() for resource deallocation.
94 # Attempting to _step() or _reset() the statement after a failed _reset() will
95 # always return SQLITE_NOMEM.
98 # Other active SQL statements:
100 # The effect of a malloc failure on concurrently executing SQL statements,
101 # particularly when the statement is executing with READ_UNCOMMITTED set and
102 # the malloc() failure mandates statement rollback only. Currently, if
103 # transaction rollback is required, all other vdbe's are aborted.
105 # Non-transient mallocs in btree.c:
106 # * The Btree structure itself
107 # * Each BtCursor structure
109 # Mallocs in pager.c:
110 # readMasterJournal() - Space to read the master journal name
111 # pager_delmaster() - Space for the entire master journal file
113 # sqlite3pager_open() - The pager structure itself
114 # sqlite3_pagerget() - Space for a new page
115 # pager_open_journal() - Pager.aInJournal[] bitmap
116 # sqlite3pager_write() - For in-memory databases only: history page and
117 # statement history page.
118 # pager_stmt_begin() - Pager.aInStmt[] bitmap
120 # None of the above are a huge problem. The most troublesome failures are the
121 # transient malloc() calls in btree.c, which can occur during the tree-balance
122 # operation. This means the tree being balanced will be internally inconsistent
123 # after the malloc() fails. To avoid the corrupt tree being read by a
124 # READ_UNCOMMITTED query, we have to make sure the transaction or statement
125 # rollback occurs before sqlite3_step() returns, not during a subsequent
126 # sqlite3_finalize().
127 #--------------------------------------------------------------------------
129 #--------------------------------------------------------------------------
130 # NOTES ON TEST IMPLEMENTATION
132 # The tests in this file are implemented differently from those in other
133 # files. Instead, tests are specified using three primitives: SQL, PREP and
134 # TEST. Each primitive has a single argument. Primitives are processed in
135 # the order they are specified in the file.
137 # A TEST primitive specifies a TCL script as its argument. When a TEST
138 # directive is encountered the Tcl script is evaluated. Usually, this Tcl
139 # script contains one or more calls to [do_test].
141 # A PREP primitive specifies an SQL script as its argument. When a PREP
142 # directive is encountered the SQL is evaluated using database connection
145 # The SQL primitives are where the action happens. An SQL primitive must
146 # contain a single, valid SQL statement as its argument. When an SQL
147 # primitive is encountered, it is evaluated one or more times to test the
148 # behaviour of the system when malloc() fails during preparation or
149 # execution of said statement. The Nth time the statement is executed,
150 # the Nth malloc is said to fail. The statement is executed until it
151 # succeeds, i.e. (M+1) times, where M is the number of mallocs() required
152 # to prepare and execute the statement.
154 # Each time an SQL statement fails, the driver program (see proc [run_test]
155 # below) figures out if a transaction has been automatically rolled back.
156 # If not, it executes any TEST block immediately proceeding the SQL
157 # statement, then reexecutes the SQL statement with the next value of N.
159 # If a transaction has been automatically rolled back, then the driver
160 # program executes all the SQL specified as part of SQL or PREP primitives
161 # between the current SQL statement and the most recent "BEGIN". Any
162 # TEST block immediately proceeding the SQL statement is evaluated, and
163 # then the SQL statement reexecuted with the incremented N value.
165 # That make any sense? If not, read the code in [run_test] and it might.
167 # Extra restriction imposed by the implementation:
169 # * If a PREP block starts a transaction, it must finish it.
170 # * A PREP block may not close a transaction it did not start.
172 #--------------------------------------------------------------------------
175 # These procs are used to build up a "program" in global variable
176 # ::run_test_script. At the end of this file, the proc [run_test] is used
177 # to execute the program (and all test cases contained therein).
179 set ::run_test_sql_id 0
180 set ::run_test_script [list]
181 proc TEST {id t} {lappend ::run_test_script -test [list $id $t]}
182 proc PREP {p} {lappend ::run_test_script -prep [string trim $p]}
183 proc DEBUG {s} {lappend ::run_test_script -debug $s}
187 # SQL ?-norollback? <sql-text>
189 # Add an 'SQL' primitive to the program (see notes above). If the -norollback
190 # switch is present, then the statement is not allowed to automatically roll
191 # back any active transaction if malloc() fails. It must rollback the statement
194 proc SQL {a1 {a2 ""}} {
195 # An SQL primitive parameter is a list of three elements, an id, a boolean
196 # value indicating if the statement may cause transaction rollback when
197 # malloc() fails, and the sql statement itself.
198 set id [incr ::run_test_sql_id]
200 lappend ::run_test_script -sql [list $id true [string trim $a1]]
202 lappend ::run_test_script -sql [list $id false [string trim $a2]]
208 # A shorthand test to see if a transaction is active or not. The first
209 # argument - $id - is the integer number of the test case. The second
210 # argument is either 1 or 0, the expected value of the auto-commit flag.
212 proc TEST_AUTOCOMMIT {id a} {
213 TEST $id "do_test \$testid { sqlite3_get_autocommit \$::DB } {$a}"
216 #--------------------------------------------------------------------------
217 # Start of test program declaration
221 # Warm body test. A malloc() fails in the middle of a CREATE TABLE statement
222 # in a single-statement transaction on an empty database. Not too much can go
227 execsql {SELECT tbl_name FROM sqlite_master;}
231 CREATE TABLE IF NOT EXISTS abc(a, b, c);
235 execsql {SELECT tbl_name FROM sqlite_master;}
239 # Insert a couple of rows into the table. each insert is in its own
240 # transaction. test that the table is unpopulated before running the inserts
241 # (and hence after each failure of the first insert), and that it has been
242 # populated correctly after the final insert succeeds.
246 execsql {SELECT * FROM abc}
249 SQL {INSERT INTO abc VALUES(1, 2, 3);}
250 SQL {INSERT INTO abc VALUES(4, 5, 6);}
251 SQL {INSERT INTO abc VALUES(7, 8, 9);}
254 execsql {SELECT * FROM abc}
255 } {1 2 3 4 5 6 7 8 9}
258 # Test a CREATE INDEX statement. Because the table 'abc' is so small, the index
259 # will all fit on a single page, so this doesn't test too much that the CREATE
260 # TABLE statement didn't test. A few of the transient malloc()s in btree.c
263 SQL {CREATE INDEX abc_i ON abc(a, b, c);}
267 SELECT * FROM abc ORDER BY a DESC;
269 } {7 8 9 4 5 6 1 2 3}
272 # Test a DELETE statement. Also create a trigger and a view, just to make sure
273 # these statements don't have any obvious malloc() related bugs in them. Note
274 # that the test above will be executed each time the DELETE fails, so we're
275 # also testing rollback of a DELETE from a table with an index on it.
277 SQL {DELETE FROM abc WHERE a > 2;}
278 SQL {CREATE TRIGGER abc_t AFTER INSERT ON abc BEGIN SELECT 'trigger!'; END;}
279 SQL {CREATE VIEW abc_v AS SELECT * FROM abc;}
283 SELECT name, tbl_name FROM sqlite_master ORDER BY name;
286 } {abc abc abc_i abc abc_t abc abc_v abc_v 1 2 3}
290 BEGIN;DELETE FROM abc;
292 for {set i 1} {$i < 100} {incr i} {
294 set b "String value $i"
295 set c [string repeat X $i]
296 append sql "INSERT INTO abc VALUES ($a, '$b', '$c');"
302 DELETE FROM abc WHERE oid IN (SELECT oid FROM abc ORDER BY random() LIMIT 5);
306 execsql {SELECT count(*) FROM abc}
311 (oid == a) AND 'String value ' || a == b AND a == length(c)
317 DELETE FROM abc WHERE oid IN (SELECT oid FROM abc ORDER BY random() LIMIT 5);
321 execsql {SELECT count(*) FROM abc}
326 (oid == a) AND 'String value ' || a == b AND a == length(c)
332 DELETE FROM abc WHERE oid IN (SELECT oid FROM abc ORDER BY random() LIMIT 5);
336 execsql {SELECT count(*) FROM abc}
341 (oid == a) AND 'String value ' || a == b AND a == length(c)
347 set padding [string repeat X 500]
350 CREATE TABLE abc(a PRIMARY KEY, padding, b, c);
351 INSERT INTO abc VALUES(0, '$padding', 2, 2);
352 INSERT INTO abc VALUES(3, '$padding', 5, 5);
353 INSERT INTO abc VALUES(6, '$padding', 8, 8);
358 execsql {SELECT a, b, c FROM abc}
359 } {0 2 2 3 5 5 6 8 8}
363 SQL {INSERT INTO abc VALUES(9, 'XXXXX', 11, 12);}
365 SQL -norollback {UPDATE abc SET a = a + 1, c = c + 1;}
367 SQL {DELETE FROM abc WHERE a = 10;}
373 sqlite3_get_autocommit $::DB
376 execsql {SELECT a, b, c FROM abc}
377 } {1 2 3 4 5 6 7 8 9}
382 CREATE TABLE abc(a, padding, b, c);
383 INSERT INTO abc VALUES(1, '$padding', 2, 3);
384 INSERT INTO abc VALUES(4, '$padding', 5, 6);
385 INSERT INTO abc VALUES(7, '$padding', 8, 9);
386 CREATE INDEX abc_i ON abc(a, padding, b, c);
390 db eval {PRAGMA cache_size = 10}
394 SQL -norllbck {INSERT INTO abc (oid, a, padding, b, c) SELECT NULL, * FROM abc}
397 execsql {SELECT a, count(*) FROM abc GROUP BY a;}
400 SQL -norllbck {INSERT INTO abc (oid, a, padding, b, c) SELECT NULL, * FROM abc}
403 execsql {SELECT a, count(*) FROM abc GROUP BY a;}
406 SQL -norllbck {INSERT INTO abc (oid, a, padding, b, c) SELECT NULL, * FROM abc}
409 execsql {SELECT a, count(*) FROM abc GROUP BY a;}
412 SQL -norllbck {INSERT INTO abc (oid, a, padding, b, c) SELECT NULL, * FROM abc}
415 execsql {SELECT a, count(*) FROM abc GROUP BY a;}
421 execsql {SELECT a, count(*) FROM abc GROUP BY a;}
426 SQL {DELETE FROM abc WHERE oid %2}
429 execsql {SELECT a, count(*) FROM abc GROUP BY a;}
432 SQL {DELETE FROM abc}
435 execsql {SELECT * FROM abc}
441 execsql {SELECT a, count(*) FROM abc GROUP BY a;}
445 # Test some schema modifications inside of a transaction. These should all
446 # cause transaction rollback if they fail. Also query a view, to cover a bit
449 PREP {DROP VIEW abc_v;}
453 SELECT name, tbl_name FROM sqlite_master;
455 } {abc abc abc_i abc}
458 SQL {CREATE TABLE def(d, e, f);}
459 SQL {CREATE TABLE ghi(g, h, i);}
463 SELECT name, tbl_name FROM sqlite_master;
465 } {abc abc abc_i abc def def ghi ghi}
467 SQL {CREATE VIEW v1 AS SELECT * FROM def, ghi}
468 SQL {CREATE UNIQUE INDEX ghi_i1 ON ghi(g);}
472 SELECT name, tbl_name FROM sqlite_master;
474 } {abc abc abc_i abc def def ghi ghi v1 v1 ghi_i1 ghi}
476 SQL {INSERT INTO def VALUES('a', 'b', 'c')}
477 SQL {INSERT INTO def VALUES(1, 2, 3)}
478 SQL -norollback {INSERT INTO ghi SELECT * FROM def}
482 SELECT * FROM def, ghi WHERE d = g;
484 } {a b c a b c 1 2 3 1 2 3}
490 SELECT * FROM v1 WHERE d = g;
492 } {a b c a b c 1 2 3 1 2 3}
495 # Test a simple multi-file transaction
499 SQL {ATTACH 'test2.db' AS aux;}
501 SQL {CREATE TABLE aux.tbl2(x, y, z)}
502 SQL {INSERT INTO tbl2 VALUES(1, 2, 3)}
503 SQL {INSERT INTO def VALUES(4, 5, 6)}
507 SELECT * FROM tbl2, def WHERE d = x;
515 SELECT * FROM tbl2, def WHERE d = x;
521 # Test what happens when a malloc() fails while there are other active
522 # statements. This changes the way sqlite3VdbeHalt() works.
524 if {![info exists ::STMT32]} {
525 set sql "SELECT name FROM sqlite_master"
526 set ::STMT32 [sqlite3_prepare $::DB $sql -1 DUMMY]
528 sqlite3_step $::STMT32
535 execsql {SELECT * FROM ghi}
539 -- There is a unique index on ghi(g), so this statement may not cause
540 -- an automatic ROLLBACK. Hence the "-norollback" switch.
541 INSERT INTO ghi SELECT '2'||g, h, i FROM ghi;
544 if {[info exists ::STMT32]} {
546 sqlite3_finalize $::STMT32
554 # End of test program declaration
555 #--------------------------------------------------------------------------
557 proc run_test {arglist iRepeat {pcstart 0} {iFailStart 1}} {
558 if {[llength $arglist] %2} {
559 error "Uneven number of arguments to TEST"
562 for {set i 0} {$i < $pcstart} {incr i} {
563 set k2 [lindex $arglist [expr {2 * $i}]]
564 set v2 [lindex $arglist [expr {2 * $i + 1}]]
565 set ac [sqlite3_get_autocommit $::DB] ;# Auto-Commit
567 -sql {db eval [lindex $v2 2]}
571 set nac [sqlite3_get_autocommit $::DB] ;# New Auto-Commit
572 if {$ac && !$nac} {set begin_pc $i}
575 db rollback_hook [list incr ::rollback_hook_count]
577 set iFail $iFailStart
579 while {$pc*2 < [llength $arglist]} {
580 # Fetch the current instruction type and payload.
581 set k [lindex $arglist [expr {2 * $pc}]]
582 set v [lindex $arglist [expr {2 * $pc + 1}]]
584 # Id of this iteration:
585 set iterid "pc=$pc.iFail=$iFail$k"
590 foreach {id script} $v {}
591 set testid "malloc3-(test $id).$iterid"
597 set ::rollback_hook_count 0
600 set testid "malloc3-(integrity $id).$iterid"
602 set ac [sqlite3_get_autocommit $::DB] ;# Auto-Commit
603 sqlite3_memdebug_fail $iFail -repeat 0
604 set rc [catch {db eval [lindex $v 2]} msg] ;# True error occurs
605 set nac [sqlite3_get_autocommit $::DB] ;# New Auto-Commit
607 if {$rc != 0 && $nac && !$ac} {
608 # Before [db eval] the auto-commit flag was clear. Now it
609 # is set. Since an error occurred we assume this was not a
610 # commit - therefore a rollback occurred. Check that the
611 # rollback-hook was invoked.
612 do_test malloc3-rollback_hook_count.$iterid {
613 set ::rollback_hook_count
617 set nFail [sqlite3_memdebug_fail -1 -benigncnt nBenign]
619 # Successful execution of sql. The number of failed malloc()
620 # calls should be equal to the number of benign failures.
621 # Otherwise a malloc() failed and the error was not reported.
623 set expr {$nFail!=$nBenign}
625 error "Unreported malloc() failure, test \"$testid\", $expr"
629 # Before the [db eval] the auto-commit flag was set, now it
630 # is clear. We can deduce that a "BEGIN" statement has just
631 # been successfully executed.
637 integrity_check $testid
638 } elseif {[regexp {.*out of memory} $msg] || [db errorcode] == 3082} {
639 # Out of memory error, as expected.
641 integrity_check $testid
644 if {![lindex $v 1] && [db errorcode] != 3082} {
645 # error "Statement \"[lindex $v 2]\" caused a rollback"
648 for {set i $begin_pc} {$i < $pc} {incr i} {
649 set k2 [lindex $arglist [expr {2 * $i}]]
650 set v2 [lindex $arglist [expr {2 * $i + 1}]]
653 -sql {set catchupsql [lindex $v2 2]}
654 -prep {set catchupsql $v2}
663 # back up to the previous "-test" block.
664 while {[lindex $arglist [expr {2 * ($pc - 1)}]] == "-test"} {
679 default { error "Unknown switch: $k" }
684 # Turn off the Tcl interface's prepared statement caching facility. Then
685 # run the tests with "persistent" malloc failures.
686 sqlite3_extended_result_codes db 1
688 run_test $::run_test_script 1
690 # Close and reopen the db.
692 forcedelete test.db test.db-journal test2.db test2.db-journal
694 sqlite3_extended_result_codes db 1
695 set ::DB [sqlite3_connection_pointer db]
697 # Turn off the Tcl interface's prepared statement caching facility in
698 # the new connnection. Then run the tests with "transient" malloc failures.
700 run_test $::run_test_script 0
702 sqlite3_memdebug_fail -1