A lexer that takes an InputRange of slices from the input.
This lexer tries to merge the speed of direct slicing with the low memory requirements
of ranges. Its input is a range whose elements are chunks of the input data; this
lexer returns slices of the original chunks, unless the output is split between two
chunks. If that's the case, a new array is allocated and returned. The various chunks
may have different sizes.
The bigger the chunks are, the better is the performance and higher the memory usage,
so finding the correct tradeoff is crucial for maximum performance. This lexer is
suitable for very large files, which are read chunk by chunk from the file system.
A lexer that takes an InputRange of slices from the input.
This lexer tries to merge the speed of direct slicing with the low memory requirements of ranges. Its input is a range whose elements are chunks of the input data; this lexer returns slices of the original chunks, unless the output is split between two chunks. If that's the case, a new array is allocated and returned. The various chunks may have different sizes.
The bigger the chunks are, the better is the performance and higher the memory usage, so finding the correct tradeoff is crucial for maximum performance. This lexer is suitable for very large files, which are read chunk by chunk from the file system.