Delay-loaded series

The DelayedSeries type provides an efficient way to create series whose data is loaded on-demand. For example, you may have a large time series stored in a CSV file or in a database and you do not want to load all the data in memory if the user only needs a small part of it.

When you create a delayed series, you specify the overall range of the series (i.e. the minimum and maximum key value) and you provide a function that loads a specified sub-range of the series. When the user accesses a continuous range of the series, the loading function is called to retrieve the data.

Creating a delayed series

To create a delayed series, we need a function that generates data for a given range. The following function generates a series with random data for a given date range with a day frequency:

let generate (low:DateTime) (high:DateTime) : seq<KeyValuePair<DateTime,float>> = 
    let rnd = Random()
    let days = int (high - low).TotalDays
    seq [ for d in 0 .. days -> KeyValuePair(low.AddDays(float d), rnd.NextDouble()) ]

Now we use DelayedSeries.FromValueLoader to create a delayed series. It takes the overall minimum and maximum key of the series and a function that loads data for a sub-range. The loading function gets the lower and upper bound as a tuple of (key, BoundaryBehavior) values where BoundaryBehavior is either Inclusive or Exclusive:

let min = DateTime(2010, 1, 1)
let max = DateTime(2013, 1, 1)

let ls = DelayedSeries.FromValueLoader(min, max, fun (lo, lob) (hi, hib) -> async {
    printfn "Query: %A - %A" lo hi
    let lo = if lob = BoundaryBehavior.Inclusive then lo else lo.AddDays(1.0)
    let hi = if hib = BoundaryBehavior.Inclusive then hi else hi.AddDays(-1.0)
    return generate lo hi })

The key thing about the above is that, so far, no data has been loaded. The loading function is called only when we access part of the series.

Slicing and using delayed series

We can now use the series as usual - for example, to get data for the entire year 2012:

let slice = ls.[DateTime(2012, 1, 1) .. DateTime(2012, 12, 31)]
slice

val slice: Series<DateTime,float> =
  
(Delayed series [01/01/2012 .. 12/31/2012]) 

val it: Series<DateTime,float> =
  
(Delayed series [01/01/2012 .. 12/31/2012])

Similarly, we can add the delayed series to a data frame. When doing this, Deedle will only load the data that is needed. In the following example, we add the series to a frame and then access only a slice:

let df = frame ["Values" => ls]
let slicedDf = df.Rows.[DateTime(2012,6,1) .. DateTime(2012,6,30)]
slicedDf

Query: 01/01/2010 00:00:00 - 01/01/2013 00:00:00
Query: 06/01/2012 00:00:00 - 06/30/2012 00:00:00
val df: Frame<DateTime,string> =
  
              Values               
01/01/2010 -> 0.9791247891456165   
01/02/2010 -> 0.48442050486447397  
01/03/2010 -> 0.055046114100890886 
01/04/2010 -> 0.8277073759186414   
01/05/2010 -> 0.4148770723370646   
01/06/2010 -> 0.21924485528638837  
01/07/2010 -> 0.8794866671434798   
01/08/2010 -> 0.01646887564415811  
01/09/2010 -> 0.9851418109117754   
01/10/2010 -> 0.9197845649999975   
01/11/2010 -> 0.7469846046082718   
01/12/2010 -> 0.6458433319801822   
01/13/2010 -> 0.12143892097310782  
01/14/2010 -> 0.7548418776335746   
01/15/2010 -> 0.020525763224742977 
:             ...                  
12/18/2012 -> 0.8116643891789911   
12/19/2012 -> 0.3929728329736314   
12/20/2012 -> 0.792918301109978    
12/21/2012 -> 0.24866410548923656  
12/22/2012 -> 0.3861025930098415   
12/23/2012 -> 0.9808474607343761   
12/24/2012 -> 0.014129521713543203 
12/25/2012 -> 0.08337495736797418  
12/26/2012 -> 0.06871703855658928  
12/27/2012 -> 0.7453997358752494   
12/28/2012 -> 0.414027069360253    
12/29/2012 -> 0.20386457309049744  
12/30/2012 -> 0.9085664043942169   
12/31/2012 -> 0.3414842429694869   
01/01/2013 -> 0.5903362139916939   

val slicedDf: Frame<DateTime,string> =
  
              Values               
06/01/2012 -> 0.8996761580334756   
06/02/2012 -> 0.7354713451025324   
06/03/2012 -> 0.6817864169881277   
06/04/2012 -> 0.2728454077746758   
06/05/2012 -> 0.2755550066482356   
06/06/2012 -> 0.914291010413889    
06/07/2012 -> 0.6646384399133223   
06/08/2012 -> 0.7306127469407832   
06/09/2012 -> 0.17263420027837417  
06/10/2012 -> 0.7042585732351456   
06/11/2012 -> 0.8339946216230908   
06/12/2012 -> 0.39939104470342157  
06/13/2012 -> 0.6759730857098085   
06/14/2012 -> 0.6873238563911674   
06/15/2012 -> 0.6304437950554188   
06/16/2012 -> 0.6016094604337836   
06/17/2012 -> 0.9182392309014247   
06/18/2012 -> 0.8355460939782494   
06/19/2012 -> 0.6986416234144435   
06/20/2012 -> 0.015196808101563097 
06/21/2012 -> 0.9683594677709381   
06/22/2012 -> 0.5153963171526164   
06/23/2012 -> 0.47035004358235066  
06/24/2012 -> 0.8192280002218137   
06/25/2012 -> 0.12654071059938987  
06/26/2012 -> 0.747251385243731    
06/27/2012 -> 0.8520421640844184   
06/28/2012 -> 0.8154801748524357   
06/29/2012 -> 0.8366022162944382   
06/30/2012 -> 0.7252097560538255   

val it: Frame<DateTime,string> =
  
              Values               
06/01/2012 -> 0.8996761580334756   
06/02/2012 -> 0.7354713451025324   
06/03/2012 -> 0.6817864169881277   
06/04/2012 -> 0.2728454077746758   
06/05/2012 -> 0.2755550066482356   
06/06/2012 -> 0.914291010413889    
06/07/2012 -> 0.6646384399133223   
06/08/2012 -> 0.7306127469407832   
06/09/2012 -> 0.17263420027837417  
06/10/2012 -> 0.7042585732351456   
06/11/2012 -> 0.8339946216230908   
06/12/2012 -> 0.39939104470342157  
06/13/2012 -> 0.6759730857098085   
06/14/2012 -> 0.6873238563911674   
06/15/2012 -> 0.6304437950554188   
06/16/2012 -> 0.6016094604337836   
06/17/2012 -> 0.9182392309014247   
06/18/2012 -> 0.8355460939782494   
06/19/2012 -> 0.6986416234144435   
06/20/2012 -> 0.015196808101563097 
06/21/2012 -> 0.9683594677709381   
06/22/2012 -> 0.5153963171526164   
06/23/2012 -> 0.47035004358235066  
06/24/2012 -> 0.8192280002218137   
06/25/2012 -> 0.12654071059938987  
06/26/2012 -> 0.747251385243731    
06/27/2012 -> 0.8520421640844184   
06/28/2012 -> 0.8154801748524357   
06/29/2012 -> 0.8366022162944382   
06/30/2012 -> 0.7252097560538255

namespace System

namespace System.Collections

namespace System.Collections.Generic

namespace Deedle

namespace Deedle.Indices

val fsi: FSharp.Compiler.Interactive.InteractiveSession

member FSharp.Compiler.Interactive.InteractiveSession.AddPrinter: ('T -> string) -> unit

val o: obj

type obj = Object

val iface: Type

Object.GetType() : Type

val fmt: Reflection.MethodInfo

Type.GetMethod(name: string) : Reflection.MethodInfo
   (+0 other overloads)
Type.GetMethod(name: string, types: Type array) : Reflection.MethodInfo
   (+0 other overloads)
Type.GetMethod(name: string, bindingAttr: Reflection.BindingFlags) : Reflection.MethodInfo
   (+0 other overloads)
Type.GetMethod(name: string, types: Type array, modifiers: Reflection.ParameterModifier array) : Reflection.MethodInfo
   (+0 other overloads)
Type.GetMethod(name: string, bindingAttr: Reflection.BindingFlags, types: Type array) : Reflection.MethodInfo
   (+0 other overloads)
Type.GetMethod(name: string, genericParameterCount: int, types: Type array) : Reflection.MethodInfo
   (+0 other overloads)
Type.GetMethod(name: string, genericParameterCount: int, types: Type array, modifiers: Reflection.ParameterModifier array) : Reflection.MethodInfo
   (+0 other overloads)
Type.GetMethod(name: string, genericParameterCount: int, bindingAttr: Reflection.BindingFlags, types: Type array) : Reflection.MethodInfo
   (+0 other overloads)
Type.GetMethod(name: string, bindingAttr: Reflection.BindingFlags, binder: Reflection.Binder, types: Type array, modifiers: Reflection.ParameterModifier array) : Reflection.MethodInfo
   (+0 other overloads)
Type.GetMethod(name: string, bindingAttr: Reflection.BindingFlags, binder: Reflection.Binder, callConvention: Reflection.CallingConventions, types: Type array, modifiers: Reflection.ParameterModifier array) : Reflection.MethodInfo
   (+0 other overloads)

Reflection.MethodBase.Invoke(obj: obj, parameters: obj array) : obj
Reflection.MethodBase.Invoke(obj: obj, invokeAttr: Reflection.BindingFlags, binder: Reflection.Binder, parameters: obj array, culture: Globalization.CultureInfo) : obj

Multiple items
val string: value: 'T -> string

--------------------
type string = String

val generate: low: DateTime -> high: DateTime -> KeyValuePair<DateTime,float> seq

val low: DateTime

Multiple items
type DateTime = new: date: DateOnly * time: TimeOnly -> unit + 16 overloads member Add: value: TimeSpan -> DateTime member AddDays: value: float -> DateTime member AddHours: value: float -> DateTime member AddMicroseconds: value: float -> DateTime member AddMilliseconds: value: float -> DateTime member AddMinutes: value: float -> DateTime member AddMonths: months: int -> DateTime member AddSeconds: value: float -> DateTime member AddTicks: value: int64 -> DateTime ...
<summary>Represents an instant in time, typically expressed as a date and time of day.</summary>

--------------------
DateTime ()
   (+0 other overloads)
DateTime(ticks: int64) : DateTime
   (+0 other overloads)
DateTime(date: DateOnly, time: TimeOnly) : DateTime
   (+0 other overloads)
DateTime(ticks: int64, kind: DateTimeKind) : DateTime
   (+0 other overloads)
DateTime(date: DateOnly, time: TimeOnly, kind: DateTimeKind) : DateTime
   (+0 other overloads)
DateTime(year: int, month: int, day: int) : DateTime
   (+0 other overloads)
DateTime(year: int, month: int, day: int, calendar: Globalization.Calendar) : DateTime
   (+0 other overloads)
DateTime(year: int, month: int, day: int, hour: int, minute: int, second: int) : DateTime
   (+0 other overloads)
DateTime(year: int, month: int, day: int, hour: int, minute: int, second: int, kind: DateTimeKind) : DateTime
   (+0 other overloads)
DateTime(year: int, month: int, day: int, hour: int, minute: int, second: int, calendar: Globalization.Calendar) : DateTime
   (+0 other overloads)

val high: DateTime

Multiple items
val seq: sequence: 'T seq -> 'T seq

--------------------
type 'T seq = IEnumerable<'T>

Multiple items
type KeyValuePair = static member Create<'TKey,'TValue> : key: 'TKey * value: 'TValue -> KeyValuePair<'TKey,'TValue>
<summary>Creates instances of the <see cref="T:System.Collections.Generic.KeyValuePair`2" /> struct.</summary>

--------------------
type KeyValuePair<'TKey,'TValue> = new: key: 'TKey * value: 'TValue -> unit member Deconstruct: key: byref<'TKey> * value: byref<'TValue> -> unit member ToString: unit -> string member Key: 'TKey member Value: 'TValue
<summary>Defines a key/value pair that can be set or retrieved.</summary>
<typeparam name="TKey">The type of the key.</typeparam>
<typeparam name="TValue">The type of the value.</typeparam>

--------------------
KeyValuePair ()
KeyValuePair(key: 'TKey, value: 'TValue) : KeyValuePair<'TKey,'TValue>

Multiple items
val float: value: 'T -> float (requires member op_Explicit)

--------------------
type float = Double

--------------------
type float<'Measure> = float

val rnd: Random

Multiple items
type Random = new: unit -> unit + 1 overload member GetHexString: stringLength: int * ?lowercase: bool -> string + 1 overload member GetItems<'T> : choices: ReadOnlySpan<'T> * length: int -> 'T array + 2 overloads member GetString: choices: ReadOnlySpan<char> * length: int -> string member Next: unit -> int + 2 overloads member NextBytes: buffer: byte array -> unit + 1 overload member NextDouble: unit -> float member NextInt64: unit -> int64 + 2 overloads member NextSingle: unit -> float32 member Shuffle<'T> : values: Span<'T> -> unit + 1 overload ...
<summary>Represents a pseudo-random number generator, which is an algorithm that produces a sequence of numbers that meet certain statistical requirements for randomness.</summary>

--------------------
Random() : Random
Random(Seed: int) : Random

val days: int

Multiple items
val int: value: 'T -> int (requires member op_Explicit)

--------------------
type int = int32

--------------------
type int<'Measure> = int

val d: int

DateTime.AddDays(value: float) : DateTime

Random.NextDouble() : float

val min: DateTime

val max: DateTime

val ls: Series<DateTime,float>

type DelayedSeries = static member FromIndexVectorLoader: scheme: IAddressingScheme * vectorBuilder: IVectorBuilder * indexBuilder: IIndexBuilder * min: 'K * max: 'K * loader: Func<'K,BoundaryBehavior,'K,BoundaryBehavior,Task<IIndex<'K> * IVector<'V>>> -> Series<'K,'V> (requires equality) + 1 overload static member FromValueLoader: min: 'K * max: 'K * loader: Func<'K,BoundaryBehavior,'K,BoundaryBehavior,Task<KeyValuePair<'K,'V> seq>> -> Series<'K,'V> (requires comparison) + 1 overload
<summary> This type exposes a single static method `DelayedSeries.Create` that can be used for constructing data series (of type <c>Series<K, V></c>) with lazily loaded data. You can use this functionality to create series that represents e.g. an entire price history in a database, but only loads data that are actually needed. For more information see the [lazy data loading tutorial](../lazysource.html). </summary>
<example> Assuming we have a function <c>generate lo hi</c> that generates data in the specified <c>DateTime</c> range, we can create lazy series as follows: <code> let ls = DelayedSeries.Create(min, max, fun (lo, lob) (hi, hib) -> async { printfn "Query: %A - %A" (lo, lob) (hi, hib) return generate lo hi }) </code> The arguments <c>min</c> and <c>max</c> specify the complete range of the series. The function passed to <c>Create</c> is called with minimal and maximal required key (<c>lo</c> and <c>hi</c>) and with two values that specify boundary behaviour. </example>
<category>Specialized frame and series types</category>

static member DelayedSeries.FromValueLoader: min: 'K * max: 'K * loader: ('K * BoundaryBehavior -> 'K * BoundaryBehavior -> Async<KeyValuePair<'K,'V> seq>) -> Series<'K,'V> (requires comparison)
static member DelayedSeries.FromValueLoader: min: 'K * max: 'K * loader: Func<'K,BoundaryBehavior,'K,BoundaryBehavior,Threading.Tasks.Task<KeyValuePair<'K,'V> seq>> -> Series<'K,'V> (requires comparison)

val lo: DateTime

val lob: BoundaryBehavior

val hi: DateTime

val hib: BoundaryBehavior

val async: AsyncBuilder

val printfn: format: Printf.TextWriterFormat<'T> -> 'T

type BoundaryBehavior = | Inclusive | Exclusive
<summary> Specifies the boundary behavior for the `IIndexBuilder.GetRange` operation (whether the boundary elements should be included or not) </summary>

union case BoundaryBehavior.Inclusive: BoundaryBehavior

val slice: Series<DateTime,float>

val df: Frame<DateTime,string>

val frame: columns: ('a * #ISeries<'c>) seq -> Frame<'c,'a> (requires equality and equality)
<summary> A function for constructing data frame from a sequence of name - column pairs. This provides a nicer syntactic sugar for `Frame.ofColumns`. </summary>
<example> To create a simple frame with two columns, you can write: <code> frame [ "A" => series [ 1 => 30.0; 2 => 35.0 ] "B" => series [ 1 => 30.0; 3 => 40.0 ] ] </code></example>
<category>Frame construction</category>

val slicedDf: Frame<DateTime,string>

property Frame.Rows: RowSeries<DateTime,string> with get
<category>Accessors and slicing</category>