Delay-loaded series

The DelayedSeries type provides an efficient way to create series whose data is loaded on-demand. For example, you may have a large time series stored in a CSV file or in a database and you do not want to load all the data in memory if the user only needs a small part of it.

When you create a delayed series, you specify the overall range of the series (i.e. the minimum and maximum key value) and you provide a function that loads a specified sub-range of the series. When the user accesses a continuous range of the series, the loading function is called to retrieve the data.

Creating a delayed series

To create a delayed series, we need a function that generates data for a given range. The following function generates a series with random data for a given date range with a day frequency:

let generate (low:DateTime) (high:DateTime) : seq<KeyValuePair<DateTime,float>> = 
    let rnd = Random()
    let days = int (high - low).TotalDays
    seq [ for d in 0 .. days -> KeyValuePair(low.AddDays(float d), rnd.NextDouble()) ]

Now we use DelayedSeries.FromValueLoader to create a delayed series. It takes the overall minimum and maximum key of the series and a function that loads data for a sub-range. The loading function gets the lower and upper bound as a tuple of (key, BoundaryBehavior) values where BoundaryBehavior is either Inclusive or Exclusive:

let min = DateTime(2010, 1, 1)
let max = DateTime(2013, 1, 1)

let ls = DelayedSeries.FromValueLoader(min, max, fun (lo, lob) (hi, hib) -> async {
    printfn "Query: %A - %A" lo hi
    let lo = if lob = BoundaryBehavior.Inclusive then lo else lo.AddDays(1.0)
    let hi = if hib = BoundaryBehavior.Inclusive then hi else hi.AddDays(-1.0)
    return generate lo hi })

The key thing about the above is that, so far, no data has been loaded. The loading function is called only when we access part of the series.

Slicing and using delayed series

We can now use the series as usual - for example, to get data for the entire year 2012:

let slice = ls.[DateTime(2012, 1, 1) .. DateTime(2012, 12, 31)]
slice

val slice: Series<DateTime,float> =
  
(Delayed series [01/01/2012 .. 12/31/2012]) 

val it: Series<DateTime,float> =
  
(Delayed series [01/01/2012 .. 12/31/2012])

Similarly, we can add the delayed series to a data frame. When doing this, Deedle will only load the data that is needed. In the following example, we add the series to a frame and then access only a slice:

let df = frame ["Values" => ls]
let slicedDf = df.Rows.[DateTime(2012,6,1) .. DateTime(2012,6,30)]
slicedDf

Query: 01/01/2010 00:00:00 - 01/01/2013 00:00:00
Query: 06/01/2012 00:00:00 - 06/30/2012 00:00:00
val df: Frame<DateTime,string> =
  
              Values               
01/01/2010 -> 0.4630553674309442   
01/02/2010 -> 0.6403382775047904   
01/03/2010 -> 0.28242924679491976  
01/04/2010 -> 0.21060959328518725  
01/05/2010 -> 0.7598702935862942   
01/06/2010 -> 0.11341434200725353  
01/07/2010 -> 0.14041083425897416  
01/08/2010 -> 0.8269469804650759   
01/09/2010 -> 0.514415157472967    
01/10/2010 -> 0.5427449000201212   
01/11/2010 -> 0.4917172366934114   
01/12/2010 -> 0.4924321058097272   
01/13/2010 -> 0.28582402441671884  
01/14/2010 -> 0.10894146039537245  
01/15/2010 -> 0.7599955336723102   
:             ...                  
12/18/2012 -> 0.4343632862271286   
12/19/2012 -> 0.9888084467953916   
12/20/2012 -> 0.9190019400409671   
12/21/2012 -> 0.34768901763037363  
12/22/2012 -> 0.9855182019625292   
12/23/2012 -> 0.4795882557779868   
12/24/2012 -> 0.1192805805986592   
12/25/2012 -> 0.8693508871846294   
12/26/2012 -> 0.9294609952209668   
12/27/2012 -> 0.8431259917498019   
12/28/2012 -> 0.023157244371258434 
12/29/2012 -> 0.29487313084862554  
12/30/2012 -> 0.00412416533907789  
12/31/2012 -> 0.5911567100470322   
01/01/2013 -> 0.26751792341990155  

val slicedDf: Frame<DateTime,string> =
  
              Values              
06/01/2012 -> 0.6034293124559077  
06/02/2012 -> 0.6804480595947768  
06/03/2012 -> 0.8845826557031836  
06/04/2012 -> 0.44080364113133896 
06/05/2012 -> 0.223055562502131   
06/06/2012 -> 0.6406546507774268  
06/07/2012 -> 0.9452072136141986  
06/08/2012 -> 0.23180956673707365 
06/09/2012 -> 0.23476394853129268 
06/10/2012 -> 0.2586971543687391  
06/11/2012 -> 0.6386733062265415  
06/12/2012 -> 0.3771661089156034  
06/13/2012 -> 0.06247864349720389 
06/14/2012 -> 0.8306454174054159  
06/15/2012 -> 0.8609633889409761  
06/16/2012 -> 0.7940079254196228  
06/17/2012 -> 0.01631475974321195 
06/18/2012 -> 0.9974648441218746  
06/19/2012 -> 0.07057429788005865 
06/20/2012 -> 0.3694506167988717  
06/21/2012 -> 0.5592714788711375  
06/22/2012 -> 0.575622316403503   
06/23/2012 -> 0.7586055322771963  
06/24/2012 -> 0.24678861640083383 
06/25/2012 -> 0.33123579870697994 
06/26/2012 -> 0.47161478159066483 
06/27/2012 -> 0.05047954037846103 
06/28/2012 -> 0.7154823670572747  
06/29/2012 -> 0.9905134074628557  
06/30/2012 -> 0.3873812635704619  

val it: Frame<DateTime,string> =
  
              Values              
06/01/2012 -> 0.6034293124559077  
06/02/2012 -> 0.6804480595947768  
06/03/2012 -> 0.8845826557031836  
06/04/2012 -> 0.44080364113133896 
06/05/2012 -> 0.223055562502131   
06/06/2012 -> 0.6406546507774268  
06/07/2012 -> 0.9452072136141986  
06/08/2012 -> 0.23180956673707365 
06/09/2012 -> 0.23476394853129268 
06/10/2012 -> 0.2586971543687391  
06/11/2012 -> 0.6386733062265415  
06/12/2012 -> 0.3771661089156034  
06/13/2012 -> 0.06247864349720389 
06/14/2012 -> 0.8306454174054159  
06/15/2012 -> 0.8609633889409761  
06/16/2012 -> 0.7940079254196228  
06/17/2012 -> 0.01631475974321195 
06/18/2012 -> 0.9974648441218746  
06/19/2012 -> 0.07057429788005865 
06/20/2012 -> 0.3694506167988717  
06/21/2012 -> 0.5592714788711375  
06/22/2012 -> 0.575622316403503   
06/23/2012 -> 0.7586055322771963  
06/24/2012 -> 0.24678861640083383 
06/25/2012 -> 0.33123579870697994 
06/26/2012 -> 0.47161478159066483 
06/27/2012 -> 0.05047954037846103 
06/28/2012 -> 0.7154823670572747  
06/29/2012 -> 0.9905134074628557  
06/30/2012 -> 0.3873812635704619

namespace System

namespace System.Collections

namespace System.Collections.Generic

namespace Deedle

namespace Deedle.Indices

val fsi: FSharp.Compiler.Interactive.InteractiveSession

member FSharp.Compiler.Interactive.InteractiveSession.AddPrinter: ('T -> string) -> unit

val o: obj

type obj = Object

val iface: Type

Object.GetType() : Type

val fmt: Reflection.MethodInfo

Type.GetMethod(name: string) : Reflection.MethodInfo
   (+0 other overloads)
Type.GetMethod(name: string, types: Type array) : Reflection.MethodInfo
   (+0 other overloads)
Type.GetMethod(name: string, bindingAttr: Reflection.BindingFlags) : Reflection.MethodInfo
   (+0 other overloads)
Type.GetMethod(name: string, types: Type array, modifiers: Reflection.ParameterModifier array) : Reflection.MethodInfo
   (+0 other overloads)
Type.GetMethod(name: string, bindingAttr: Reflection.BindingFlags, types: Type array) : Reflection.MethodInfo
   (+0 other overloads)
Type.GetMethod(name: string, genericParameterCount: int, types: Type array) : Reflection.MethodInfo
   (+0 other overloads)
Type.GetMethod(name: string, genericParameterCount: int, types: Type array, modifiers: Reflection.ParameterModifier array) : Reflection.MethodInfo
   (+0 other overloads)
Type.GetMethod(name: string, genericParameterCount: int, bindingAttr: Reflection.BindingFlags, types: Type array) : Reflection.MethodInfo
   (+0 other overloads)
Type.GetMethod(name: string, bindingAttr: Reflection.BindingFlags, binder: Reflection.Binder, types: Type array, modifiers: Reflection.ParameterModifier array) : Reflection.MethodInfo
   (+0 other overloads)
Type.GetMethod(name: string, bindingAttr: Reflection.BindingFlags, binder: Reflection.Binder, callConvention: Reflection.CallingConventions, types: Type array, modifiers: Reflection.ParameterModifier array) : Reflection.MethodInfo
   (+0 other overloads)

Reflection.MethodBase.Invoke(obj: obj, parameters: obj array) : obj
Reflection.MethodBase.Invoke(obj: obj, invokeAttr: Reflection.BindingFlags, binder: Reflection.Binder, parameters: obj array, culture: Globalization.CultureInfo) : obj

Multiple items
val string: value: 'T -> string

--------------------
type string = String

val generate: low: DateTime -> high: DateTime -> KeyValuePair<DateTime,float> seq

val low: DateTime

Multiple items
type DateTime = new: date: DateOnly * time: TimeOnly -> unit + 16 overloads member Add: value: TimeSpan -> DateTime member AddDays: value: float -> DateTime member AddHours: value: float -> DateTime member AddMicroseconds: value: float -> DateTime member AddMilliseconds: value: float -> DateTime member AddMinutes: value: float -> DateTime member AddMonths: months: int -> DateTime member AddSeconds: value: float -> DateTime member AddTicks: value: int64 -> DateTime ...
<summary>Represents an instant in time, typically expressed as a date and time of day.</summary>

--------------------
DateTime ()
   (+0 other overloads)
DateTime(ticks: int64) : DateTime
   (+0 other overloads)
DateTime(date: DateOnly, time: TimeOnly) : DateTime
   (+0 other overloads)
DateTime(ticks: int64, kind: DateTimeKind) : DateTime
   (+0 other overloads)
DateTime(date: DateOnly, time: TimeOnly, kind: DateTimeKind) : DateTime
   (+0 other overloads)
DateTime(year: int, month: int, day: int) : DateTime
   (+0 other overloads)
DateTime(year: int, month: int, day: int, calendar: Globalization.Calendar) : DateTime
   (+0 other overloads)
DateTime(year: int, month: int, day: int, hour: int, minute: int, second: int) : DateTime
   (+0 other overloads)
DateTime(year: int, month: int, day: int, hour: int, minute: int, second: int, kind: DateTimeKind) : DateTime
   (+0 other overloads)
DateTime(year: int, month: int, day: int, hour: int, minute: int, second: int, calendar: Globalization.Calendar) : DateTime
   (+0 other overloads)

val high: DateTime

Multiple items
val seq: sequence: 'T seq -> 'T seq

--------------------
type 'T seq = IEnumerable<'T>

Multiple items
type KeyValuePair = static member Create<'TKey,'TValue> : key: 'TKey * value: 'TValue -> KeyValuePair<'TKey,'TValue>
<summary>Creates instances of the <see cref="T:System.Collections.Generic.KeyValuePair`2" /> struct.</summary>

--------------------
type KeyValuePair<'TKey,'TValue> = new: key: 'TKey * value: 'TValue -> unit member Deconstruct: key: byref<'TKey> * value: byref<'TValue> -> unit member ToString: unit -> string member Key: 'TKey member Value: 'TValue
<summary>Defines a key/value pair that can be set or retrieved.</summary>
<typeparam name="TKey">The type of the key.</typeparam>
<typeparam name="TValue">The type of the value.</typeparam>

--------------------
KeyValuePair ()
KeyValuePair(key: 'TKey, value: 'TValue) : KeyValuePair<'TKey,'TValue>

Multiple items
val float: value: 'T -> float (requires member op_Explicit)

--------------------
type float = Double

--------------------
type float<'Measure> = float

val rnd: Random

Multiple items
type Random = new: unit -> unit + 1 overload member GetHexString: stringLength: int * ?lowercase: bool -> string + 1 overload member GetItems<'T> : choices: ReadOnlySpan<'T> * length: int -> 'T array + 2 overloads member GetString: choices: ReadOnlySpan<char> * length: int -> string member Next: unit -> int + 2 overloads member NextBytes: buffer: byte array -> unit + 1 overload member NextDouble: unit -> float member NextInt64: unit -> int64 + 2 overloads member NextSingle: unit -> float32 member Shuffle<'T> : values: Span<'T> -> unit + 1 overload ...
<summary>Represents a pseudo-random number generator, which is an algorithm that produces a sequence of numbers that meet certain statistical requirements for randomness.</summary>

--------------------
Random() : Random
Random(Seed: int) : Random

val days: int

Multiple items
val int: value: 'T -> int (requires member op_Explicit)

--------------------
type int = int32

--------------------
type int<'Measure> = int

val d: int

DateTime.AddDays(value: float) : DateTime

Random.NextDouble() : float

val min: DateTime

val max: DateTime

val ls: Series<DateTime,float>

type DelayedSeries = static member FromIndexVectorLoader: scheme: IAddressingScheme * vectorBuilder: IVectorBuilder * indexBuilder: IIndexBuilder * min: 'K * max: 'K * loader: Func<'K,BoundaryBehavior,'K,BoundaryBehavior,Task<IIndex<'K> * IVector<'V>>> -> Series<'K,'V> (requires equality) + 1 overload static member FromValueLoader: min: 'K * max: 'K * loader: Func<'K,BoundaryBehavior,'K,BoundaryBehavior,Task<KeyValuePair<'K,'V> seq>> -> Series<'K,'V> (requires comparison) + 1 overload
<summary> This type exposes a single static method `DelayedSeries.Create` that can be used for constructing data series (of type <c>Series<K, V></c>) with lazily loaded data. You can use this functionality to create series that represents e.g. an entire price history in a database, but only loads data that are actually needed. For more information see the [lazy data loading tutorial](../lazysource.html). </summary>
<example> Assuming we have a function <c>generate lo hi</c> that generates data in the specified <c>DateTime</c> range, we can create lazy series as follows: <code> let ls = DelayedSeries.Create(min, max, fun (lo, lob) (hi, hib) -> async { printfn "Query: %A - %A" (lo, lob) (hi, hib) return generate lo hi }) </code> The arguments <c>min</c> and <c>max</c> specify the complete range of the series. The function passed to <c>Create</c> is called with minimal and maximal required key (<c>lo</c> and <c>hi</c>) and with two values that specify boundary behaviour. </example>
<category>Specialized frame and series types</category>

static member DelayedSeries.FromValueLoader: min: 'K * max: 'K * loader: ('K * BoundaryBehavior -> 'K * BoundaryBehavior -> Async<KeyValuePair<'K,'V> seq>) -> Series<'K,'V> (requires comparison)
static member DelayedSeries.FromValueLoader: min: 'K * max: 'K * loader: Func<'K,BoundaryBehavior,'K,BoundaryBehavior,Threading.Tasks.Task<KeyValuePair<'K,'V> seq>> -> Series<'K,'V> (requires comparison)

val lo: DateTime

val lob: BoundaryBehavior

val hi: DateTime

val hib: BoundaryBehavior

val async: AsyncBuilder

val printfn: format: Printf.TextWriterFormat<'T> -> 'T

type BoundaryBehavior = | Inclusive | Exclusive
<summary> Specifies the boundary behavior for the `IIndexBuilder.GetRange` operation (whether the boundary elements should be included or not) </summary>

union case BoundaryBehavior.Inclusive: BoundaryBehavior

val slice: Series<DateTime,float>

val df: Frame<DateTime,string>

val frame: columns: ('a * #ISeries<'c>) seq -> Frame<'c,'a> (requires equality and equality)
<summary> A function for constructing data frame from a sequence of name - column pairs. This provides a nicer syntactic sugar for `Frame.ofColumns`. </summary>
<example> To create a simple frame with two columns, you can write: <code> frame [ "A" => series [ 1 => 30.0; 2 => 35.0 ] "B" => series [ 1 => 30.0; 3 => 40.0 ] ] </code></example>
<category>Frame construction</category>

val slicedDf: Frame<DateTime,string>

property Frame.Rows: RowSeries<DateTime,string> with get
<category>Accessors and slicing</category>