This paste expires on 2023-04-25 18:27:27.547566. Repaste, or download this paste. . Pasted through v1-api.

14:24 [snoopjedi@denton ~]
$ python3 -m tokenize <<< "'this string is a single token'"
1,0-1,31:           STRING         "'this string is a single token'"
1,31-1,32:          NEWLINE        '\n'
2,0-2,0:            ENDMARKER      ''
14:27 [snoopjedi@denton ~]
$ python3 -m tokenize <<< "['this list', 'is not']"
1,0-1,1:            OP             '['
1,1-1,12:           STRING         "'this list'"
1,12-1,13:          OP             ','
1,14-1,22:          STRING         "'is not'"
1,22-1,23:          OP             ']'
1,23-1,24:          NEWLINE        '\n'
2,0-2,0:            ENDMARKER      ''
Filename: out.txt. Size: 604b. View raw, , hex, or download this file.