Public Git Hosting - pcreposix-compat.git/blob

1 PCRE POSIX Compat

2 =================

3

4 What Is It?

5 -----------

6

7 POSIX compatibility updates for the "pcreposix" functionality provided with

8 the PCRE library so that "pcreposix.h" and the corresponding library may be

9 used as a standard POSIX "regex.h" substitute.

10

11 In addition `pcre_jit_exec` stubs are included when building with

12 `--disable-jit` and when building with `--enable-jit` `pcre_jit_exec` will

13 automatically fall back to `pcre_exec` when no JIT compilation is available.

14 This makes the library work more seamlessly whether in JIT mode or not (perhaps

15 because JIT cannot handle the specific pattern or because the platform in

16 question does not allow creation of writable plus executable memory for the JIT

17 compilation) even when callers are not using the JIT API entirely correctly.

18

19 See the "How to Use" section at the bottom if you're having a tl;dr moment.

20

21

22 Wherefore Art Thou?

23 -------------------

24

25 Some poorly implemented regex engines go bananas over this:

26

27 w++++++++++++++++++++++

28

29 Technically that's not a POSIX ERE (using `+` requires an ERE not a BRE).

30

31 But this is:

32

33 (((((((((((((((((((((w+)+)+)+)+)+)+)+)+)+)+)+)+)+)+)+)+)+)+)+)+)+

34

35 You need only compile it with one of those engines like so:

36

37 grep -E '(((((((((((((((((((((w+)+)+)+)+)+)+)+)+)+)+)+)+)+)+)+)+)+)+)+)+)+'

38

39 to watch the bananas hit the fan.

40

41 PCRE does not suffer bananas easily and with a few, relatively simple,

42 enhancements can be used as a substitute for the POSIX "regex.h" `regcomp`

43 and `regexec` functions to avoid going bananas over nothing.

44

45

46 Good Timing Results for pcregrep

47 --------------------------------

48

49 The PCRE distribution contains a `pcregrep` utility that works very much

50 like `grep` except that all patterns are PCRE patterns (as long as `-F` is

51 not used).

52

53 Here are some timing results for the problem regex shown above when passed

54 to `pcregrep`. The "count" value along the "x" axis represents how many

55 "+" quantifiers are being used in the pattern.

56

57 Some of these are running in VMs which is why these "good" timings are in

58 the tens of milliseconds rather than much closer to 0.

59

60 These timing results confirm that using PCRE does not go bananas on these

61 kinds of patterns. These are all "good" results.

62

63 Testing "pcregrep" on Linux 3.16.0-4-amd64 x86_64

64 |0.030| * *

65 | | = =

66 |0.027| = =

67 | | = =

68 |0.024| = =

69 | | = =

70 |0.021| = =

71 | | = = * * *

72 |0.018| = = = = =

73 | | = = = = =

74 |0.015| = = = = =

75 | | = = = = =

76 |0.012| = = = = =

77 | | = * = * * * * * * = * * = * * * = * * * * * * * * *

78 |0.009| = = = = = = = = = = = = = = = = = = = = = = = = = =

79 | | = = = = = = = = = = = = = = = = = = = = = = = = = =

80 |0.006| = = = = = = = = = = = = = = = = = = = = = = = = = =

81 | | = = = = = = = = = = = = = = = = = = = = = = = = = =

82 |0.003| = = = = = = = = = = = = = = = = = = = = = = = = = =

83 | | = = = = = = = = = = = = = = = = = = = = = = = = = =

84 |0.000| = * = = = = = = * = = = = = = = = = = = * = = = = * = = = =

85 | sec +------------------------------------------------------------

86 |:::::| 1 2 3 4 5 6 7 8 9 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 2 3

87 |:::::| 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0

88

89 Testing "pcregrep" on FreeBSD 10.3-RELEASE-p4 amd64

90 |0.031| *

91 | | =

92 |0.029| =

93 | | =

94 |0.027| =

95 | | =

96 |0.024| =

97 | | = * * * * * *

98 |0.022| = = = = = = =

99 | | = = = = = = =

100 |0.020| = = = = = = =

101 | | = = = = = = =

102 |0.017| = = = = = = =

103 | | * * * = * = * = * * * = * * * * * = * = * * * * * = *

104 |0.015| = = = = = = = = = = = = = = = = = = = = = = = = = = =

105 | | = = = = = = = = = = = = = = = = = = = = = = = = = = =

106 |0.013| = = = = = = = = = = = = = = = = = = = = = = = = = = =

107 | | = = = = = = = = = = = = = = = = = = = = = = = = = = =

108 |0.010| = = = = = = = = = = = = = = = = = = = = = = = = = = =

109 | | = = = = = = = = = = = = = = = = = = = = = = = = = = =

110 |0.008| = = = = = = = = = = = * = * = = = = = = = * = = = = = = = =

111 | sec +------------------------------------------------------------

112 |:::::| 1 2 3 4 5 6 7 8 9 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 2 3

113 |:::::| 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0

114

115 Testing "pcregrep" on Darwin 9.8.0 i386

116 |0.010| * * * * * *

117 | | = = = = = =

118 |0.009| = = = = = =

119 | | = = = = = =

120 |0.008| = = = = = =

121 | | = = = = = =

122 |0.007| = = = = = =

123 | | = = = = = =

124 |0.006| = = = = = =

125 | | = = = = = =

126 |0.005| = = = = = =

127 | | = = = = = =

128 |0.004| = = = = = =

129 | | = = = = = =

130 |0.003| = = = = = =

131 | | = = = = = =

132 |0.002| = = = = = =

133 | | = = = = = =

134 |0.001| = = = = = =

135 | | = = = = = =

136 |0.000| * * * * = * * * * * = * * * * * = * * * * = * = * * * * * =

137 | sec +------------------------------------------------------------

138 |:::::| 1 2 3 4 5 6 7 8 9 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 2 3

139 |:::::| 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0

140

141

142 Good and Bad grep Timing Results

143 --------------------------------

144

145 There's only one "good" result here, but it should be obvious which one

146 it is. That there's only one "good" result here instead of two will

147 probably come as somewhat of a surprise to some folks, especially once

148 the next section's results are viewed as well.

149

150 Testing "grep -E" on Linux 3.16.0-4-amd64 x86_64

151 |2.370| *

152 | | =

153 |2.133| =

154 | | =

155 |1.896| =

156 | | =

157 |1.659| =

158 | | =

159 |1.422| =

160 | | =

161 |1.185| * =

162 | | = =

163 |0.948| = =

164 | | = =

165 |0.711| = =

166 | | = =

167 |0.474| * = =

168 | | = = =

169 |0.237| * = = =

170 | | * * = = = =

171 |0.000| * * * * * * * * * * = = = = = =

172 | sec +--------------------------------

173 |:::::| 1 2 3 4 5 6 7 8 9 1 1 1 1 1 1 1

174 |:::::| 0 1 2 3 4 5 6

175

176 Testing "grep -E" on FreeBSD 10.3-RELEASE-p4 amd64

177 |8.820| *

178 | | =

179 |7.939| =

180 | | =

181 |7.058| =

182 | | =

183 |6.177| =

184 | | =

185 |5.295| =

186 | | =

187 |4.414| =

188 | | =

189 |3.533| =

190 | | =

191 |2.652| =

192 | | =

193 |1.770| =

194 | | * =

195 |0.889| * = =

196 | | * = = =

197 |0.008| * * * * * * * * * * * * = = = =

198 | sec +--------------------------------

199 |:::::| 1 2 3 4 5 6 7 8 9 1 1 1 1 1 1 1

200 |:::::| 0 1 2 3 4 5 6

201

202 Testing "grep -E" on Darwin 9.8.0 i386

203 |0.020| *

204 | | =

205 |0.018| =

206 | | =

207 |0.016| =

208 | | =

209 |0.014| =

210 | | =

211 |0.012| =

212 | | =

213 |0.010| * * = *

214 | | = = = =

215 |0.008| = = = =

216 | | = = = =

217 |0.006| = = = =

218 | | = = = =

219 |0.004| = = = =

220 | | = = = =

221 |0.002| = = = =

222 | | = = = =

223 |0.000| * * * * * = * * * * * * = * * * * * * = * * * * * * = * * *

224 | sec +------------------------------------------------------------

225 |:::::| 1 2 3 4 5 6 7 8 9 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 2 3

226 |:::::| 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0

227

228

229 Good and Bad git Timing Results

230 -------------------------------

231

232 There's only one "good" result here, but it should be obvious which one

233 it is (and, no, it's not the same one as in the previous set). Again,

234 it's a bit surprising there's only one "good" result here.

235

236 Testing "git log --grep" on Linux 3.16.0-4-amd64 x86_64

237 |2.330| *

238 | | =

239 |2.097| =

240 | | =

241 |1.864| =

242 | | =

243 |1.631| =

244 | | =

245 |1.398| =

246 | | =

247 |1.165| * =

248 | | = =

249 |0.932| = =

250 | | = =

251 |0.699| = =

252 | | * = =

253 |0.466| = = =

254 | | = = =

255 |0.233| * = = =

256 | | * = = = =

257 |0.000| * * * * * * * * * * * = = = = =

258 | sec +--------------------------------

259 |:::::| 1 2 3 4 5 6 7 8 9 1 1 1 1 1 1 1

260 |:::::| 0 1 2 3 4 5 6

261

262 Testing "git log --grep" on FreeBSD 10.3-RELEASE-p4 amd64

263 |0.047| *

264 | | =

265 |0.044| =

266 | | =

267 |0.041| =

268 | | = *

269 |0.038| = =

270 | | = =

271 |0.034| = =

272 | | = =

273 |0.031| * * * * * * * * * = * * * * =

274 | | = = = = = = = = = = = = = = =

275 |0.028| = = = = = = = = = = = = = = =

276 | | = = = = = = = = = = = = = = =

277 |0.025| = = = = = = = = = = = = = = =

278 | | * * = * = = = = = * = * = * = * * * * = = = * * = = * =

279 |0.022| = = = = = = = = = = = = = = = = = = = = = = = = = = = =

280 | | = = = = = = = = = = = = = = = = = = = = = = = = = = = =

281 |0.019| = = = = = = = = = = = = = = = = = = = = = = = = = = = =

282 | | = = = = = = = = = = = = = = = = = = = = = = = = = = = =

283 |0.016| = = = = = = * = = = = = = = * = = = = = = = = = = = = = = =

284 | sec +------------------------------------------------------------

285 |:::::| 1 2 3 4 5 6 7 8 9 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 2 3

286 |:::::| 0 1 2 3 4 5 6 7 8 9 0 1 2 3 4 5 6 7 8 9 0

287

288 Testing "git log --grep" on Darwin 9.8.0 i386

289 |3.150| *

290 | | =

291 |2.835| =

292 | | =

293 |2.520| =

294 | | =

295 |2.205| =

296 | | =

297 |1.890| =

298 | | =

299 |1.575| =

300 | | * =

301 |1.260| = =

302 | | = =

303 |0.945| = =

304 | | = =

305 |0.630| * = =

306 | | = = =

307 |0.315| * = = =

308 | | * = = = =

309 |0.000| * * * * * * * * * * * * = = = = =

310 | sec +----------------------------------

311 |:::::| 1 2 3 4 5 6 7 8 9 1 1 1 1 1 1 1 1

312 |:::::| 0 1 2 3 4 5 6 7

313

314

315 PCRE POSIX Means What?

316 ----------------------

317

318 With a name like "pcreposix" it could be taken to mean either of these:

319

320 1. PCRE used as an implementation backend to provide POSIX semantics

321 2. PCRE semantics provided via a POSIX-similar API

322

323 If you guessed (1), you're wrong. At least historically speaking, "pcreposix"

324 has always meant (2) which can be a bit confusing since it provided neither

325 Basic Regular Expression (BRE) support nor Extended Regular Expression (ERE)

326 support via the `regcomp` and `regexec` functions in the "pcreposix.h" header,

327 but has "posix" in its name and POSIX `regcomp` and `regexec` provide those.

328

329 What it did provide was Perl Compatible Regular Expression (PCRE) support via

330 that header.

331

332

333 PCRE POSIX Compat

334 -----------------

335

336 These compatibility updates for "pcreposix" update it to provide _both_ (1)

337 and (2) by introducing full support for POSIX BREs and EREs and providing a

338 new `REG_PCRE` option for the `regcomp` function to activate PCRE support.

339

340 As a bonus a `REG_JAVASCPT` option is also provided to activate PCRE's

341 Javascript Regular Expression (JRE) support.

342

343 All in one place with one API you get your BREs, EREs, PCREs and JREs.

344

345 In addition, if the library is built with `--enable-jit` and the platform

346 allows creation of writable plus executable memory for JIT compilation, the JIT

347 pattern compiler will automatically be used by `regcomp` when available no

348 matter what options are passed to the `regcomp` function.

349

350

351 Backwards Compatibility

352 -----------------------

353

354 Not with older "pcreposix" libraries. The original version of "pcreposix.h"

355 defines the `REG_EXTENDED` constant to be 0. This makes backwards

356 compatibility for the "pcreposix" library impossible.

357

358 The version for the "pcreposix" library has been changed from a previous

359 `-version-info 0:4:0` to the current `-version-info 1:0:0`. What this means

360 is that both the old and new "pcreposix" libraries can coexist on the same

361 system. The old, original one will have a name like "libpcreposix.0.<ext>"

362 while the new one will be named like "libpcreposix.1.<ext>".

363

364 The "pcreposix.h" header has also seen substantial changes and while an attempt

365 has been made to make sure same-named constants keep the same value between the

366 original version of "pcreposix.h" and the new one, in some cases that's just

367 not possible. For example, `REG_EXTENDED` must not be 0.

368

369 To address this issue and also facilitate use as a "regex.h" substitute, the

370 new "pcreposix.h" header and a new "regex.h" header (it's the same header) are

371 now installed to "$includedir/pcreposix/pcreposix.h" and

372 "$includedir/pcreposix/regex.h" whereas the original header was installed

373 simply to "$includedir/pcreposix.h".

374

375 So again, the old and new versions of the "pcreposix.h" header can coexist on

376 the same system. But now, to use the new "pcreposix" as a drop-in "regex.h"

377 substitute all that's required is adding a "-I $includedir/pcreposix" option

378 to the compile line and "-lpcreposix -lpcre" options to the link line.

379

380 Although if "libpcreposix.<ext>" is pointing at the old "libpcreposix.0.<ext>"

381 library then that will have to be "-lpcreposix.1" instead.

382

383 The version of PCRE remains unchanged because:

384

385 1. The only client of the new options is the new pcreposix

386 2. Versions of PCRE not supporting the new options will complain about them

387 causing the new pcreposix `regcomp` implementation to return an error

388 3. Even if non-pcreposix clients make use of the newly provided PCRE options

389 regex's compiled using them can be executed by any version of PCRE

390

391

392 Slight of Hand

393 --------------

394

395 So how does PCRE now, all of a sudden, support POSIX-semantics BREs and EREs?

396

397 Magic. A slight of hand. "Syntactic Sugar".

398

399 When the pattern is compiled in "BRE" mode, automagical virtual backslash

400 escaping and unescaping takes place to transform the incoming BRE into an

401 equivalent PCRE.

402

403 To a lesser extent the same thing happens in "ERE" mode but really only in

404 the one case (`[^foo]` does not match `\n` when one of the new options is

405 enabled). But again, more slight of hand (an extra `\n` is virtually inserted

406 into the negated character class to get the desired semantics).

407

408 There are five new PCRE options in all. (The other three also perform more

409 slight-of-hand to provide POSIX `REG_EXTENDED`, BSD `REG_PEND` and BSD

410 `REG_NOSPEC` compatibility.)

411

412 The bottom line is that saving off a compiled PCRE regular expression that used

413 one (or more) of the new options, loading it on a system without support for

414 the new options and "exec"ing it works and matches what it's supposed to.

415

416 The rest of the POSIX semantics compatibility is provided by having the new

417 "pcreposix" library manipulate other PCRE options (`PCRE_DOTALL`,

418 `PCRE_DOLLARENDOLY` and `PCRE_MULTILINE`) based on the options supplied to

419 the `regcomp` function.

420

421 Full details are in the "pcreposix.h" header, but basically, sticking to only

422 POSIX-defined options results in POSIX semantics. Using either the new

423 `REG_PCRE` or `REG_JAVASCPT` non-POSIX options gets you either PCREs or JREs

424 respectively.

425

426 The "*BSD" compatibility options (`REG_PEND`, `REG_NOSPEC` and `REG_STARTEND`)

427 are supported in all regex modes (but `REG_NOSPEC` acts like its own mode).

428

429

430 New pcreposix Old pcre

431 ----------------------

432

433 So what happens if a new "pcreposix" library is linked against an old "pcre"

434 library?

435

436 It depends.

437

438 If the "pcreposix" client has set `REG_PCRE` and/or `REG_JAVASCPT` and avoids

439 using `REG_NOSPEC` and `REG_PEND` then everything else will work.

440

441 Alternatively, if the "pcreposix" client sets `REG_EXTENDED` and does NOT set

442 any of `REG_NEWLINE`, `REG_NOSPEC` or `REG_PEND` it will also still work.

443

444 If none of `REG_EXTENDED`, `REG_PCRE` or `REG_JAVASCPT` are set or either of

445 `REG_PEND` or `REG_NOSPEC` is set it will fail at `regcomp` time with a

446 `REG_INVARG` error.

447

448 The fact that "pcreposix" non-BRE and non-REG_NEWLINE functionality works fully

449 and correctly with an "old" "-lpcre" library and returns an error rather than

450 giving incorrect results for the rest is the reason that the PCRE library

451 version remains unchanged by these compatibility updates.

452

453 An application that links against a shared "pcre" library and needs to use one

454 or more of the options that require the enhanced "pcre" library should simply

455 attempt to compile a simple pattern using the needed options at startup and

456 bail out immediately if it gets a `REG_INVARG` result.

457

458

459 What's this JIT?

460 ----------------

461

462 The PCRE library has the ability (if built with `--enable-jit`) to compile

463 pattern matching patterns directly into machine code on some platforms. This

464 "Just In Time" pattern compilation requires:

465

466 1. The PCRE library must be built with the `--enable-jit` configuration option

467 2. The platform must permit creation of `PROT_WRITE | PROT_EXEC` memory

468 3. The specific pattern and options must be supported by the JIT compiler

469 4. Extra JIT API calls must be used

470

471 The `regcomp` and `regexec` implementations included in this repository

472 automatically take care of 2-4 with automatic, silent fallback to non-JIT when

473 JIT is not supported for whatever reason (including disallowed write + exec).

474

475 Simply add the `--enable-jit` option when running `./configure` and the

476 pcreposix library will automatically use JIT pattern matching when available.

477

478 If the platform in question simply does not allow JIT at all (see [here][1] and

479 [here][2] for some discussion about this), it's more efficient to just build

480 the library with the default `--disable-jit` configuration option.

481

482 If building for Mac OS X against an older version of the Mac OS X SDK with

483 `--enable-jit` and the intent is to potentially use the resulting code on newer

484 versions of OS X, the following defines should be added at configure time:

485

486 CPPFLAGS="-DTARGET_OS_OSX=1 -DMAP_JIT=0x0800" ./configure --enable-jit ...

487

488 Newer SDKs would provide those exact values and the PCRE JIT runtime is smart

489 enough to only use `MAP_JIT` when the OS version it's running on supports it.

490

491 [1]: <https://bugs.exim.org/show_bug.cgi?id=1749>

492 [2]: <https://bugzilla.redhat.com/show_bug.cgi?id=1290432>

493

494

495 How to Use

496 ----------

497

498 Build and install like normal for PCRE (e.g. run "./configure" then "make"

499 and then "make install").

500

501 Remember to run "./configure" with `--enable-jit` if you want the JIT pattern

502 compiler support to be present (see the previous section).

503

504 This repository already contains the necessary pre-generated "configure" and

505 patches pre-applied to the PCRE 8.44 tarball release. Clone it, download a

506 tarball of it or see the "pcreposix-compat-patches" branch for individual

507 patches to apply to a PCRE 8.44 tarball yourself.

508

509 Clients that explicitly need to use non-POSIX options should include the header

510 as `#include <pcreposix/pcreposix.h>` and make sure they set the `REG_PCRE`

511 and/or `REG_JAVASCPT` option bit(s) for `regcomp` to get PCRE or JRE support.

512

513 For use as a drop-in substitute for the POSIX "regex.h", clients should just

514 continue to `#include <regex.h>` and then compile with the proper include

515 "-I $includedir/pcreposix" option.

516

517 In any case, clients of "pcreposix" must link against BOTH "-lpcreposix" and

518 "-lpcre" (wherever they happen to be).

519

520 If you wish to use this as a substitute "regex.h" when building Git, see the

521 accompanying "README-GIT-REGEX" and "config.mak" files for assistance with

522 doing that.

523

524 See the "New pcreposix Old pcre" section above for a discussion of what happens

525 when you link against the new "pcreposix" library and an older "pcre" library.

526 In some cases that may be a desirable scenario where a static version of the

527 new "pcreposix" library is linked into an application which is then linked

528 against an older shared "pcre" library. Read the "New pcreposix Old pcre"

529 section for full details.

530

531

532 License

533 -------

534

535 These updates and enhancements are licensed under the same terms as PCRE

536 itself.

537

538

539 Project Home Page

540 -----------------

541

542 https://github.com/mackyle/pcreposix-compat