testing/testcases.html

   1 <html>
   2 <head>
   3 <title>Test cases</title>
   4 </head>
   5
   6 <body>
   7 <H1>Test cases</H1>
   8 <H2>Test case results</H2>
   9 The test results are stored in seperate documents. One document for
  10 each subversion number.
  11 <table border="1">
  12        <tr><td>Test results</td><td>comment</td></tr>
  13         <tr><td><a href="examples/SAM7S256Test/results/607.html">SAM7 R607</a></td><td>PASS</td></tr>
  14         <tr><td><a href="examples/STR710Test/results/607.html">STR710 R607</a></td><td>PASS</td></tr>
  15
  16        <tr><td><a href="results/template.html">template</a></td><td>Test results template</td></tr>
  17 </table>
  18
  19 <H2>Vocabulary</H2>
  20 <table border="1">
  21        <tr>
  22                <td width="100">Passed version</td>
  23
  24                <td>The latest branch and version on which the test is known to pass</td>
  25        </tr>
  26        <tr>
  27                <td width="100">Broken version</td>
  28                <td>The latest branch and version on which the test is known to fail. n/a when older than passed version.</td>
  29        </tr>
  30        <tr>
  31                <td width="100">ID</td>
  32                <td>A unqiue ID to refer to a test. The unique numbers are maintained in this file. Note that the same test can be run on different hardware/interface. Each combination yields a unique id. </td>
  33        </tr>
  34        <tr>
  35                <td width="100">Test case</td>
  36                <td>An atomic entity that describes the operations needed to test a feature or only a part of it. The test case should:
  37                        <ul>
  38                                <li>be uniquely identifiable</li>
  39                                <li>define the complete prerequisites of the test (eg: the target, the interface, the initial state of the system)</li>
  40                                <li>define the input to be applied to the system in order to execute the test</li>
  41                                <li>define the expected output</li>
  42                                <li>contain the output resulted by running the test case</li>
  43                                <li>contain the result of the test (pass/fail)</li>
  44                        </ul>
  45                </td>
  46        </tr>
  47        <tr>
  48                <td width="100">Test suite</td>
  49                <td>A (completable) collection of test cases</td>
  50        </tr>
  51        <tr>
  52                <td width="100">Testing</td>
  53                <td>Testing refers to running the test suite for a specific revision of the software,
  54                for one or many targets, using one or many JTAG interfaces. Testing should be be stored
  55                along with all the other records for that specific revision. For releases, the results
  56                can be stored along with the binaries</td>
  57        </tr>
  58        <tr>
  59                <td width="100">Target = ANY</td>
  60                <td>Any target can be used for this test</td>
  61        </tr>
  62        <tr>
  63                <td width="100">Interface = ANY</td>
  64                <td>Any interface can be used for this test</td>
  65        </tr>
  66        <tr>
  67                <td width="100">Target = "reset_config srst_and_trst"</td>
  68                <td>Any target which supports the reset_config above</td>
  69        </tr>
  70 </table>
  71
  72 <H1>Test cases</H1>
  73
  74 <H2>Connectivity</H2>
  75 <table border=1>
  76        <tr>
  77                <td>ID</td>
  78                <td>Target</td>
  79                <td>Interface</td>
  80                <td>Description</td>
  81                <td>Initial state</td>
  82                <td>Input</td>
  83                <td>Expected output</td>
  84                <td>Pass/Fail</td>
  85        </tr>
  86        <tr>
  87                <td><a name="CON001"/>CON001</td>
  88                <td>ALL</td>
  89                <td>ALL</td>
  90                <td>Telnet connection</td>
  91                <td>Power on, jtag target attached</td>
  92                <td>On console, type<br><code>telnet ip port</code></td>
  93                <td><code>Open On-Chip Debugger<br>></code></td>
  94                <td>PASS/FAIL</td>
  95        </tr>
  96        <tr>
  97                <td><a name="CON002"/>CON002</td>
  98                <td>ALL</td>
  99                <td>ALL</td>
 100                <td>GDB server connection</td>
 101                <td>Power on, jtag target attached</td>
 102                <td>On GDB console, type<br><code>target remote ip:port</code></td>
 103                <td><code>Remote debugging using 10.0.0.73:3333</code></td>
 104                <td>PASS/FAIL</td>
 105        </tr>
 106 </table>
 107
 108 <H2>Reset</H2>
 109 <table border=1>
 110        <tr>
 111                <td>ID</td>
 112                <td>Target</td>
 113                <td>Interface</td>
 114                <td>Description</td>
 115                <td>Initial state</td>
 116                <td>Input</td>
 117                <td>Expected output</td>
 118                <td>Pass/Fail</td>
 119        </tr>
 120        <tr>
 121                <td><a name="RES001"/>RES001</td>
 122                <td>Fill in!</td>
 123                <td>Fill in!</td>
 124                <td>Reset halt on a blank target</td>
 125                <td>Erase all the content of the flash</td>
 126                <td>Connect via the telnet interface and type <br><code>reset halt</code></td>
 127                <td>Reset should return without error and the output should contain<br><code>target state: halted<br>pc = 0</code></td>
 128                <td>PASS/FAIL</td>
 129        </tr>
 130        <tr>
 131                <td><a name="RES002"/>RES002</td>
 132                <td>Fill in!</td>
 133                <td>Fill in!</td>
 134                <td>Reset init on a blank target</td>
 135                <td>Erase all the content of the flash</td>
 136                <td>Connect via the telnet interface and type <br><code>reset init</code></td>
 137                <td>Reset should return without error and the output should contain <br><code>executing reset script 'name_of_the_script'</code></td>
 138                <td>PASS/FAIL</td>
 139        </tr>
 140        <tr>
 141                <td><a name="RES003"/>RES003</td>
 142                <td>Fill in!</td>
 143                <td>Fill in!</td>
 144                <td>Reset after a power cycle of the target</td>
 145                <td>Reset the target then power cycle the target</td>
 146                <td>Connect via the telnet interface and type <br><code>reset halt</code> after the power was detected</td>
 147                <td>Reset should return without error and the output should contain<br><code>target state: halted</code></td>
 148                <td>PASS/FAIL</td>
 149        </tr>
 150        <tr>
 151                <td><a name="RES004"/>RES004</td>
 152                <td>ARM7/9,reset_config srst_and_trst</td>
 153                <td>ANY</td>
 154                <td>Reset halt on a blank target where reset halt is supported</td>
 155                <td>Erase all the content of the flash</td>
 156                <td>Connect via the telnet interface and type <br><code>reset halt</code></td>
 157                <td>Reset should return without error and the output should contain<br><code>target state: halted<br>pc = 0</code></td>
 158                <td>PASS/FAIL</td>
 159        </tr>
 160        <tr>
 161                <td><a name="RES005"/>RES005</td>
 162                <td>arm926ejs,reset_config srst_and_trst</td>
 163                <td>ANY</td>
 164                <td>Reset halt on a blank target where reset halt is supported. This target has problems with the reset vector catch being disabled by TRST</td>
 165                <td>Erase all the content of the flash</td>
 166                <td>Connect via the telnet interface and type <br><code>reset halt</code></td>
 167                <td>Reset should return without error and the output should contain<br><code>target state: halted<br>pc = 0</code></td>
 168                <td>PASS/FAIL</td>
 169        </tr>
 170 </table>
 171
 172 <H2>JTAG Speed</H2>
 173 <table border=1>
 174        <tr>
 175                <td>ID</td>
 176                <td>Target</td>
 177                <td>Interface</td>
 178                <td>Description</td>
 179                <td>Initial state</td>
 180                <td>Input</td>
 181                <td>Expected output</td>
 182                <td>Pass/Fail</td>
 183        </tr>
 184        <tr>
 185                <td><a name="SPD001"/>RES001</td>
 186                <td>Fill in!</td>
 187                <td>Fill in!</td>
 188                <td>16MHz on normal operation</td>
 189                <td>Reset init the target according to RES002 </td>
 190                <td>Exercise a memory access over the JTAG, for example <br><code>mdw 0x0 32</code></td>
 191                <td>The command should run without any errors. If any JTAG checking errors happen, the test failed</td>
 192                <td>PASS/FAIL</td>
 193        </tr>
 194 </table>
 195
 196 <H2>Debugging</H2>
 197 <table border=1>
 198        <tr>
 199                <td>ID</td>
 200                <td>Target</td>
 201                <td>Interface</td>
 202                <td>Description</td>
 203                <td>Initial state</td>
 204                <td>Input</td>
 205                <td>Expected output</td>
 206                <td>Pass/Fail</td>
 207        </tr>
 208        <tr>
 209                <td><a name="DBG001"/>DBG001</td>
 210                <td>Fill in!</td>
 211                <td>Fill in!</td>
 212                <td>Load is working</td>
 213                <td>Reset init is working, RAM is accesible, GDB server is started</td>
 214                <td>On the console of the OS: <br>
 215                        <code>arm-elf-gdb test_ram.elf</code><br>
 216                        <code>(gdb) target remote ip:port</code><br>
 217                        <code>(gdb) load</load>
 218                </td>
 219                <td>Load should return without error, typical output looks like:<br>
 220                        <code>
 221                                Loading section .text, size 0x14c lma 0x0<br>
 222                                Start address 0x40, load size 332<br>
 223                                Transfer rate: 180 bytes/sec, 332 bytes/write.<br>
 224                        </code>
 225                </td>
 226                <td>PASS/FAIL</td>
 227        </tr>
 228        <tr>
 229                <td><a name="DBG002"/>DBG002</td>
 230                <td>Fill in!</td>
 231                <td>Fill in!</td>
 232                <td>Software breakpoint</td>
 233                <td>Load the test_ram.elf application, use instructions from GDB001</td>
 234                <td>In the GDB console:<br>
 235                        <code>
 236                                (gdb) monitor arm7_9 sw_bkpts enable<br>
 237                                software breakpoints enabled<br>
 238                                (gdb) break main<br>
 239                                Breakpoint 1 at 0xec: file src/main.c, line 71.<br>
 240                                (gdb) continue<br>
 241                                Continuing.
 242                        </code>
 243                </td>
 244                <td>The software breakpoint should be reached, a typical output looks like:<br>
 245                        <code>
 246                                target state: halted<br>
 247                                target halted in ARM state due to breakpoint, current mode: Supervisor<br>
 248                                cpsr: 0x000000d3 pc: 0x000000ec<br>
 249                                <br>
 250                                Breakpoint 1, main () at src/main.c:71<br>
 251                                71        DWORD a = 1;
 252                        </code>
 253                </td>
 254                <td>PASS/FAIL</td>
 255        </tr>
 256        <tr>
 257                <td><a name="DBG003"/>DBG003</td>
 258                <td>Fill in!</td>
 259                <td>Fill in!</td>
 260                <td>Single step in a RAM application</td>
 261                <td>Load the test_ram.elf application, use instructions from GDB001, break in main using the instructions from GDB002</td>
 262                <td>In GDB, type <br><code>(gdb) step</code></td>
 263                <td>The next instruction should be reached, typical output:<br>
 264                        <code>
 265                                (gdb) step<br>
 266                                target state: halted<br>
 267                                target halted in ARM state due to single step, current mode: Abort<br>
 268                                cpsr: 0x20000097 pc: 0x000000f0<br>
 269                                target state: halted<br>
 270                                target halted in ARM state due to single step, current mode: Abort<br>
 271                                cpsr: 0x20000097 pc: 0x000000f4<br>
 272                                72        DWORD b = 2;
 273                        </code>
 274                </td>
 275                <td>PASS/FAIL</td>
 276        </tr>
 277        <tr>
 278                <td><a name="DBG004"/>DBG004</td>
 279                <td>Fill in!</td>
 280                <td>Fill in!</td>
 281                <td>Software break points are working after a reset</td>
 282                <td>Load the test_ram.elf application, use instructions from GDB001, break in main using the instructions from GDB002</td>
 283                <td>In GDB, type <br><code>
 284                        (gdb) monitor reset<br>
 285                        (gdb) load<br>
 286                        (gdb) continue<br>
 287                        </code></td>
 288                <td>The breakpoint should be reached, typical output:<br>
 289                        <code>
 290                                target state: halted<br>
 291                                target halted in ARM state due to breakpoint, current mode: Supervisor<br>
 292                                cpsr: 0x000000d3 pc: 0x000000ec<br>
 293                                <br>
 294                                Breakpoint 1, main () at src/main.c:71<br>
 295                                71        DWORD a = 1;
 296                        </code>
 297                </td>
 298                <td>PASS/FAIL</td>
 299        </tr>
 300        <tr>
 301                <td><a name="DBG005"/>DBG005</td>
 302                <td>Fill in!</td>
 303                <td>Fill in!</td>
 304                <td>Hardware breakpoint</td>
 305                <td>Flash the test_rom.elf application. Make this test after FLA004 has passed</td>
 306                <td>Be sure that <code>gdb_memory_map</code> and <code>gdb_flash_program</code> are enabled. In GDB, type <br>
 307                        <code>
 308                                (gdb) monitor reset<br>
 309                                (gdb) load<br>
 310                                Loading section .text, size 0x194 lma 0x100000<br>
 311                                Start address 0x100040, load size 404<br>
 312                                Transfer rate: 179 bytes/sec, 404 bytes/write.<br>
 313                                (gdb) monitor arm7_9  force_hw_bkpts enable<br>
 314                                force hardware breakpoints enabled<br>
 315                                (gdb) break main<br>
 316                                Breakpoint 1 at 0x100134: file src/main.c, line 69.<br>
 317                                (gdb) continue<br>
 318                        </code>
 319                </td>
 320                <td>The breakpoint should be reached, typical output:<br>
 321                        <code>
 322                                Continuing.<br>
 323                                <br>
 324                                Breakpoint 1, main () at src/main.c:69<br>
 325                                69        DWORD a = 1;<br>
 326                        </code>
 327                </td>
 328                <td>PASS/FAIL</td>
 329        </tr>
 330        <tr>
 331                <td><a name="DBG006"/>DBG006</td>
 332                <td>Fill in!</td>
 333                <td>Fill in!</td>
 334                <td>Hardware breakpoint is set after a reset</td>
 335                <td>Follow the instructions to flash and insert a hardware breakpoint from DBG005</td>
 336                <td>In GDB, type <br>
 337                        <code>
 338                                (gdb) monitor reset<br>
 339                                (gdb) monitor reg pc 0x100000<br>
 340                                pc (/32): 0x00100000<br>
 341                                (gdb) continue
 342                        </code>
 343                </td>
 344                <td>The breakpoint should be reached, typical output:<br>
 345                        <code>
 346                                Continuing.<br>
 347                                <br>
 348                                Breakpoint 1, main () at src/main.c:69<br>
 349                                69        DWORD a = 1;<br>
 350                        </code>
 351                </td>
 352                <td>PASS/FAIL</td>
 353        </tr>
 354        <tr>
 355                <td><a name="DBG007"/>DBG007</td>
 356                <td>Fill in!</td>
 357                <td>Fill in!</td>
 358                <td>Single step in ROM</td>
 359                <td>Flash the test_rom.elf application and set a breakpoint in main, use DBG005. Make this test after FLA004 has passed</td>
 360                <td>Be sure that <code>gdb_memory_map</code> and <code>gdb_flash_program</code> are enabled. In GDB, type <br>
 361                        <code>
 362                                (gdb) monitor reset<br>
 363                                (gdb) load<br>
 364                                Loading section .text, size 0x194 lma 0x100000<br>
 365                                Start address 0x100040, load size 404<br>
 366                                Transfer rate: 179 bytes/sec, 404 bytes/write.<br>
 367                                (gdb) monitor arm7_9  force_hw_bkpts enable<br>
 368                                force hardware breakpoints enabled<br>
 369                                (gdb) break main<br>
 370                                Breakpoint 1 at 0x100134: file src/main.c, line 69.<br>
 371                                (gdb) continue<br>
 372                                Continuing.<br>
 373                                <br>
 374                                Breakpoint 1, main () at src/main.c:69<br>
 375                                69        DWORD a = 1;<br>
 376                                (gdb) step
 377                        </code>
 378                </td>
 379                <td>The breakpoint should be reached, typical output:<br>
 380                        <code>
 381                                target state: halted<br>
 382                                target halted in ARM state due to single step, current mode: Supervisor<br>
 383                                cpsr: 0x60000013 pc: 0x0010013c<br>
 384                                70        DWORD b = 2;<br>
 385                        </code>
 386                </td>
 387                <td>PASS/FAIL</td>
 388        </tr>
 389 </table>
 390
 391 <H2>RAM access</H2>
 392 Note: these tests are not designed to test/debug the target, but to test functionalities!
 393 <table border=1>
 394        <tr>
 395                <td>ID</td>
 396                <td>Target</td>
 397                <td>Interface</td>
 398                <td>Description</td>
 399                <td>Initial state</td>
 400                <td>Input</td>
 401                <td>Expected output</td>
 402                <td>Pass/Fail</td>
 403        </tr>
 404        <tr>
 405                <td><a name="RAM001"/>RAM001</td>
 406                <td>Fill in!</td>
 407                <td>Fill in!</td>
 408                <td>32 bit Write/read RAM</td>
 409                <td>Reset init is working</td>
 410                <td>On the telnet interface<br>
 411                        <code>  > mww ram_address 0xdeadbeef 16<br>
 412                                        > mdw ram_address 32
 413                        </code>
 414                </td>
 415                <td>The commands should execute without error. A clear failure is a memory access exception. The result of running the commands should be a list of 16 locations 32bit long containing 0xdeadbeef.<br>
 416                        <code>
 417                                > mww 0x0 0xdeadbeef 16<br>
 418                                > mdw 0x0 32<br>
 419                                0x00000000: deadbeef deadbeef deadbeef deadbeef deadbeef deadbeef deadbeef deadbeef<br>
 420                                0x00000020: deadbeef deadbeef deadbeef deadbeef deadbeef deadbeef deadbeef deadbeef<br>
 421                                0x00000040: e1a00000 e59fa51c e59f051c e04aa000 00080017 00009388 00009388 00009388<br>
 422                                0x00000060: 00009388 0002c2c0 0002c2c0 000094f8 000094f4 00009388 00009388 00009388<br>
 423                        </code>
 424                </td>
 425                <td>PASS/FAIL</td>
 426        </tr>
 427        <tr>
 428                <td><a name="RAM001"/>RAM001</td>
 429                <td>Fill in!</td>
 430                <td>Fill in!</td>
 431                <td>16 bit Write/read RAM</td>
 432                <td>Reset init is working</td>
 433                <td>On the telnet interface<br>
 434                        <code>  > mwh ram_address 0xbeef 16<br>
 435                                        > mdh ram_address 32
 436                        </code>
 437                </td>
 438                <td>The commands should execute without error. A clear failure is a memory access exception. The result of running the commands should be a list of 16 locations 16bit long containing 0xbeef.<br>
 439                        <code>
 440                                > mwh 0x0 0xbeef 16<br>
 441                                > mdh 0x0 32<br>
 442                                0x00000000: beef beef beef beef beef beef beef beef beef beef beef beef beef beef beef beef<br>
 443                                0x00000020: 00e0 0000 021c 0000 0240 0000 026c 0000 0288 0000 0000 0000 0388 0000 0350 0000<br>
 444                                >
 445                        </code>
 446                </td>
 447                <td>PASS/FAIL</td>
 448        </tr>
 449        <tr>
 450                <td><a name="RAM003"/>RAM003</td>
 451                <td>Fill in!</td>
 452                <td>Fill in!</td>
 453                <td>8 bit Write/read RAM</td>
 454                <td>Reset init is working</td>
 455                <td>On the telnet interface<br>
 456                        <code>  > mwb ram_address 0xab 16<br>
 457                                        > mdb ram_address 32
 458                        </code>
 459                </td>
 460                <td>The commands should execute without error. A clear failure is a memory access exception. The result of running the commands should be a list of 16 locations 8bit long containing 0xab.<br>
 461                        <code>
 462                                > mwh 0x0 0x0 16<br>
 463                                > mwb ram_address 0xab 16<br>
 464                                > mdb ram_address 32<br>
 465                                0x00000000: ab ab ab ab ab ab ab ab ab ab ab ab ab ab ab ab 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00<br>
 466                                >
 467                        </code>
 468                </td>
 469                <td>PASS/FAIL</td>
 470        </tr>
 471 </table>
 472
 473
 474
 475 <H2>Flash access</H2>
 476 <table border=1>
 477        <tr>
 478                <td>ID</td>
 479                <td>Target</td>
 480                <td>Interface</td>
 481                <td>Description</td>
 482                <td>Initial state</td>
 483                <td>Input</td>
 484                <td>Expected output</td>
 485                <td>Pass/Fail</td>
 486        </tr>
 487        <tr>
 488                <td><a name="FLA002"/>FLA002</td>
 489                <td>Fill in!</td>
 490                <td>Fill in!</td>
 491                <td>flash fillw</td>
 492                <td>Reset init is working, flash is probed</td>
 493                <td>On the telnet interface<br>
 494                        <code>  > flash fillw 0x1000000 0xdeadbeef 16
 495                        </code>
 496                </td>
 497                <td>The commands should execute without error. The output looks like:<br>
 498                        <code>
 499                                wrote 64 bytes to 0x01000000 in 11.610000s (0.091516 kb/s)
 500                        </code><br>
 501                        To verify the contents of the flash:<br>
 502                        <code>
 503                                > mdw 0x1000000 32<br>
 504                                0x01000000: deadbeef deadbeef deadbeef deadbeef deadbeef deadbeef deadbeef deadbeef<br>
 505                                0x01000020: deadbeef deadbeef deadbeef deadbeef deadbeef deadbeef deadbeef deadbeef<br>
 506                                0x01000040: ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff<br>
 507                                0x01000060: ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff
 508                        </code>
 509                </td>
 510                <td>PASS/FAIL</td>
 511        </tr>
 512        <tr>
 513                <td><a name="FLA003"/>FLA003</td>
 514                <td>Fill in!</td>
 515                <td>Fill in!</td>
 516                <td>Flash erase</td>
 517                <td>Reset init is working, flash is probed</td>
 518                <td>On the telnet interface<br>
 519                        <code>  >  flash erase_address 0x1000000 0x2000
 520                        </code>
 521                </td>
 522                <td>The commands should execute without error.<br>
 523                        <code>
 524                                erased address 0x01000000 length 8192 in 4.970000s
 525                        </code>
 526                        To check that the flash has been erased, read at different addresses. The result should always be 0xff.
 527                        <code>
 528                                > mdw 0x1000000 32<br>
 529                                0x01000000: ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff<br>
 530                                0x01000020: ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff<br>
 531                                0x01000040: ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff<br>
 532                                0x01000060: ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff ffffffff
 533                        </code>
 534                </td>
 535                <td>PASS/FAIL</td>
 536        </tr>
 537        <tr>
 538                <td><a name="FLA004"/>FLA004</td>
 539                <td>Fill in!</td>
 540                <td>Fill in!</td>
 541                <td>Loading to flash from GDB</td>
 542                <td>Reset init is working, flash is probed, connectivity to GDB server is working</td>
 543                <td>Start GDB using a ROM elf image, eg: arm-elf-gdb test_rom.elf. <br>
 544                                <code>
 545                                        (gdb) target remote ip:port<br>
 546                                        (gdb) monitor reset<br>
 547                                        (gdb) load<br>
 548                                        Loading section .text, size 0x194 lma 0x100000<br>
 549                                        Start address 0x100040, load size 404<br>
 550                                        Transfer rate: 179 bytes/sec, 404 bytes/write.
 551                                        (gdb) monitor verify_image path_to_elf_file
 552                                </code>
 553                </td>
 554                <td>The output should look like:<br>
 555                        <code>
 556                                verified 404 bytes in 5.060000s
 557                        </code><br>
 558                        The failure message is something like:<br>
 559                        <code>Verify operation failed address 0x00200000. Was 0x00 instead of 0x18</code>
 560                </td>
 561                <td>PASS/FAIL</td>
 562        </tr>
 563 </table>
 564
 565 </body>
 566 </html>