Iterators

Lazy evaluation for efficiently reading large datasets.

This guide covers using iterators for efficient lazy evaluation when reading large datasets. Iterators allow you to process data in chunks without loading everything into memory.

What are Iterators?

While standard read methods cover most use cases, there are situations where you need to process more data than can fit in memory. Iterators are processed server-side, reading data in consistently sized chunks and avoiding the need to load everything into memory at once.

Opening an Iterator

To open an iterator, use the open_iterator method with a time range and channel specification.

iterator = client.open_iterator(
  start,
  end,
  ["temperature", "pressure"]
)

Chunk Size Option

By default, Synnax uses a chunk size of 100,000 samples. To configure a custom chunk size, use the chunk_size parameter with the desired number of samples per iteration.

iterator = client.open_iterator(
    start,
    end,
    "my_precise_tc",
    chunk_size=100,
)

Smaller chunk sizes reduce memory usage but may increase the number of network requests. Larger chunk sizes can improve throughput but require more memory.

Iterating through Data

Use a for loop to iterate through data in chunks. Each iteration yields a frame containing a portion of the data.

for frame in iterator:
    print(frame)

Closing the Iterator

After you’re done iterating, it’s essential to close the iterator to release the network connection and other related resources.

iterator.close()

Using structured cleanup patterns ensures the iterator is always closed, even if an exception is thrown.

# Python's context manager is the recommended approach
with client.open_iterator(start, end, "my_precise_tc") as iterator:
    for frame in iterator:
        print(frame)

# Alternatively, use a try/finally block
iterator = client.open_iterator(start, end, "my_precise_tc")
try:
    for frame in iterator:
        print(frame)
finally:
    iterator.close()