Delay-loaded series

The DelayedSeries type provides an efficient way to create series whose data is loaded on-demand. For example, you may have a large time series stored in a CSV file or in a database and you do not want to load all the data in memory if the user only needs a small part of it.

When you create a delayed series, you specify the overall range of the series (i.e. the minimum and maximum key value) and you provide a function that loads a specified sub-range of the series. When the user accesses a continuous range of the series, the loading function is called to retrieve the data.

Creating a delayed series

To create a delayed series, we need a function that generates data for a given range. The following function generates a series with random data for a given date range with a day frequency:

let generate (low:DateTime) (high:DateTime) : seq<KeyValuePair<DateTime,float>> = 
    let rnd = Random()
    let days = int (high - low).TotalDays
    seq [ for d in 0 .. days -> KeyValuePair(low.AddDays(float d), rnd.NextDouble()) ]

Now we use DelayedSeries.FromValueLoader to create a delayed series. It takes the overall minimum and maximum key of the series and a function that loads data for a sub-range. The loading function gets the lower and upper bound as a tuple of (key, BoundaryBehavior) values where BoundaryBehavior is either Inclusive or Exclusive:

let min = DateTime(2010, 1, 1)
let max = DateTime(2013, 1, 1)

let ls = DelayedSeries.FromValueLoader(min, max, fun (lo, lob) (hi, hib) -> async {
    printfn "Query: %A - %A" lo hi
    let lo = if lob = BoundaryBehavior.Inclusive then lo else lo.AddDays(1.0)
    let hi = if hib = BoundaryBehavior.Inclusive then hi else hi.AddDays(-1.0)
    return generate lo hi })

The key thing about the above is that, so far, no data has been loaded. The loading function is called only when we access part of the series.

Slicing and using delayed series

We can now use the series as usual - for example, to get data for the entire year 2012:

let slice = ls.[DateTime(2012, 1, 1) .. DateTime(2012, 12, 31)]
slice

val slice: Series<DateTime,float> =
  
(Delayed series [01/01/2012 .. 12/31/2012]) 

val it: Series<DateTime,float> =
  
(Delayed series [01/01/2012 .. 12/31/2012])

Similarly, we can add the delayed series to a data frame. When doing this, Deedle will only load the data that is needed. In the following example, we add the series to a frame and then access only a slice:

let df = frame ["Values" => ls]
let slicedDf = df.Rows.[DateTime(2012,6,1) .. DateTime(2012,6,30)]
slicedDf

Query: 01/01/2010 00:00:00 - 01/01/2013 00:00:00
Query: 06/01/2012 00:00:00 - 06/30/2012 00:00:00
val df: Frame<DateTime,string> =
  
              Values               
01/01/2010 -> 0.5267164496360556   
01/02/2010 -> 0.3720283549752236   
01/03/2010 -> 0.24470848995750116  
01/04/2010 -> 0.4481049588507323   
01/05/2010 -> 0.20159175250820505  
01/06/2010 -> 0.13395613212178137  
01/07/2010 -> 0.3489051085343726   
01/08/2010 -> 0.07368487572542415  
01/09/2010 -> 0.6214320764352411   
01/10/2010 -> 0.2758139303846353   
01/11/2010 -> 0.11978529437700847  
01/12/2010 -> 0.26291063379347057  
01/13/2010 -> 0.3009909216884854   
01/14/2010 -> 0.9789839163105012   
01/15/2010 -> 0.5087985446114097   
:             ...                  
12/18/2012 -> 0.9179815899467187   
12/19/2012 -> 0.6710383507704968   
12/20/2012 -> 0.8641788638147143   
12/21/2012 -> 0.40309132063614084  
12/22/2012 -> 0.5829829245340428   
12/23/2012 -> 0.3305811867670294   
12/24/2012 -> 0.2673116775699682   
12/25/2012 -> 0.009134509171713656 
12/26/2012 -> 0.6119123050260399   
12/27/2012 -> 0.15822071151006067  
12/28/2012 -> 0.9232057120589281   
12/29/2012 -> 0.9276304083413784   
12/30/2012 -> 0.12022817639055572  
12/31/2012 -> 0.025118466788991878 
01/01/2013 -> 0.6996108500927961   

val slicedDf: Frame<DateTime,string> =
  
              Values                
06/01/2012 -> 0.534524613699325     
06/02/2012 -> 0.0027380205514602185 
06/03/2012 -> 0.87900458391429      
06/04/2012 -> 0.021868414010187576  
06/05/2012 -> 0.5799654020043086    
06/06/2012 -> 0.44428999216532306   
06/07/2012 -> 0.8062515688107135    
06/08/2012 -> 0.3189984363951064    
06/09/2012 -> 0.013409416130576113  
06/10/2012 -> 0.34765979842122396   
06/11/2012 -> 0.8018651828659668    
06/12/2012 -> 0.9822847362219627    
06/13/2012 -> 0.07409166574402104   
06/14/2012 -> 0.8938449474486153    
06/15/2012 -> 0.7963153808714927    
06/16/2012 -> 0.06619830647941749   
06/17/2012 -> 0.2466404220384767    
06/18/2012 -> 0.45751279234264863   
06/19/2012 -> 0.3164251169261175    
06/20/2012 -> 0.296707301652977     
06/21/2012 -> 0.639484398833874     
06/22/2012 -> 0.9261427571836959    
06/23/2012 -> 0.3126542993937418    
06/24/2012 -> 0.46954303579304235   
06/25/2012 -> 0.26167622568006166   
06/26/2012 -> 0.8142245351856843    
06/27/2012 -> 0.6600672673416649    
06/28/2012 -> 0.7664079269763849    
06/29/2012 -> 0.5527022765560446    
06/30/2012 -> 0.44940417040949965   

val it: Frame<DateTime,string> =
  
              Values                
06/01/2012 -> 0.534524613699325     
06/02/2012 -> 0.0027380205514602185 
06/03/2012 -> 0.87900458391429      
06/04/2012 -> 0.021868414010187576  
06/05/2012 -> 0.5799654020043086    
06/06/2012 -> 0.44428999216532306   
06/07/2012 -> 0.8062515688107135    
06/08/2012 -> 0.3189984363951064    
06/09/2012 -> 0.013409416130576113  
06/10/2012 -> 0.34765979842122396   
06/11/2012 -> 0.8018651828659668    
06/12/2012 -> 0.9822847362219627    
06/13/2012 -> 0.07409166574402104   
06/14/2012 -> 0.8938449474486153    
06/15/2012 -> 0.7963153808714927    
06/16/2012 -> 0.06619830647941749   
06/17/2012 -> 0.2466404220384767    
06/18/2012 -> 0.45751279234264863   
06/19/2012 -> 0.3164251169261175    
06/20/2012 -> 0.296707301652977     
06/21/2012 -> 0.639484398833874     
06/22/2012 -> 0.9261427571836959    
06/23/2012 -> 0.3126542993937418    
06/24/2012 -> 0.46954303579304235   
06/25/2012 -> 0.26167622568006166   
06/26/2012 -> 0.8142245351856843    
06/27/2012 -> 0.6600672673416649    
06/28/2012 -> 0.7664079269763849    
06/29/2012 -> 0.5527022765560446    
06/30/2012 -> 0.44940417040949965

namespace System

namespace System.Collections

namespace System.Collections.Generic

namespace Deedle

namespace Deedle.Indices

val fsi: FSharp.Compiler.Interactive.InteractiveSession

member FSharp.Compiler.Interactive.InteractiveSession.AddPrinter: ('T -> string) -> unit

val o: obj

type obj = Object

val iface: Type

Object.GetType() : Type

val fmt: Reflection.MethodInfo

Type.GetMethod(name: string) : Reflection.MethodInfo
   (+0 other overloads)
Type.GetMethod(name: string, types: Type array) : Reflection.MethodInfo
   (+0 other overloads)
Type.GetMethod(name: string, bindingAttr: Reflection.BindingFlags) : Reflection.MethodInfo
   (+0 other overloads)
Type.GetMethod(name: string, types: Type array, modifiers: Reflection.ParameterModifier array) : Reflection.MethodInfo
   (+0 other overloads)
Type.GetMethod(name: string, bindingAttr: Reflection.BindingFlags, types: Type array) : Reflection.MethodInfo
   (+0 other overloads)
Type.GetMethod(name: string, genericParameterCount: int, types: Type array) : Reflection.MethodInfo
   (+0 other overloads)
Type.GetMethod(name: string, genericParameterCount: int, types: Type array, modifiers: Reflection.ParameterModifier array) : Reflection.MethodInfo
   (+0 other overloads)
Type.GetMethod(name: string, genericParameterCount: int, bindingAttr: Reflection.BindingFlags, types: Type array) : Reflection.MethodInfo
   (+0 other overloads)
Type.GetMethod(name: string, bindingAttr: Reflection.BindingFlags, binder: Reflection.Binder, types: Type array, modifiers: Reflection.ParameterModifier array) : Reflection.MethodInfo
   (+0 other overloads)
Type.GetMethod(name: string, bindingAttr: Reflection.BindingFlags, binder: Reflection.Binder, callConvention: Reflection.CallingConventions, types: Type array, modifiers: Reflection.ParameterModifier array) : Reflection.MethodInfo
   (+0 other overloads)

Reflection.MethodBase.Invoke(obj: obj, parameters: obj array) : obj
Reflection.MethodBase.Invoke(obj: obj, invokeAttr: Reflection.BindingFlags, binder: Reflection.Binder, parameters: obj array, culture: Globalization.CultureInfo) : obj

Multiple items
val string: value: 'T -> string

--------------------
type string = String

val generate: low: DateTime -> high: DateTime -> KeyValuePair<DateTime,float> seq

val low: DateTime

Multiple items
type DateTime = new: date: DateOnly * time: TimeOnly -> unit + 16 overloads member Add: value: TimeSpan -> DateTime member AddDays: value: float -> DateTime member AddHours: value: float -> DateTime member AddMicroseconds: value: float -> DateTime member AddMilliseconds: value: float -> DateTime member AddMinutes: value: float -> DateTime member AddMonths: months: int -> DateTime member AddSeconds: value: float -> DateTime member AddTicks: value: int64 -> DateTime ...
<summary>Represents an instant in time, typically expressed as a date and time of day.</summary>

--------------------
DateTime ()
   (+0 other overloads)
DateTime(ticks: int64) : DateTime
   (+0 other overloads)
DateTime(date: DateOnly, time: TimeOnly) : DateTime
   (+0 other overloads)
DateTime(ticks: int64, kind: DateTimeKind) : DateTime
   (+0 other overloads)
DateTime(date: DateOnly, time: TimeOnly, kind: DateTimeKind) : DateTime
   (+0 other overloads)
DateTime(year: int, month: int, day: int) : DateTime
   (+0 other overloads)
DateTime(year: int, month: int, day: int, calendar: Globalization.Calendar) : DateTime
   (+0 other overloads)
DateTime(year: int, month: int, day: int, hour: int, minute: int, second: int) : DateTime
   (+0 other overloads)
DateTime(year: int, month: int, day: int, hour: int, minute: int, second: int, kind: DateTimeKind) : DateTime
   (+0 other overloads)
DateTime(year: int, month: int, day: int, hour: int, minute: int, second: int, calendar: Globalization.Calendar) : DateTime
   (+0 other overloads)

val high: DateTime

Multiple items
val seq: sequence: 'T seq -> 'T seq

--------------------
type 'T seq = IEnumerable<'T>

Multiple items
type KeyValuePair = static member Create<'TKey,'TValue> : key: 'TKey * value: 'TValue -> KeyValuePair<'TKey,'TValue>
<summary>Creates instances of the <see cref="T:System.Collections.Generic.KeyValuePair`2" /> struct.</summary>

--------------------
type KeyValuePair<'TKey,'TValue> = new: key: 'TKey * value: 'TValue -> unit member Deconstruct: key: byref<'TKey> * value: byref<'TValue> -> unit member ToString: unit -> string member Key: 'TKey member Value: 'TValue
<summary>Defines a key/value pair that can be set or retrieved.</summary>
<typeparam name="TKey">The type of the key.</typeparam>
<typeparam name="TValue">The type of the value.</typeparam>

--------------------
KeyValuePair ()
KeyValuePair(key: 'TKey, value: 'TValue) : KeyValuePair<'TKey,'TValue>

Multiple items
val float: value: 'T -> float (requires member op_Explicit)

--------------------
type float = Double

--------------------
type float<'Measure> = float

val rnd: Random

Multiple items
type Random = new: unit -> unit + 1 overload member GetHexString: stringLength: int * ?lowercase: bool -> string + 1 overload member GetItems<'T> : choices: ReadOnlySpan<'T> * length: int -> 'T array + 2 overloads member GetString: choices: ReadOnlySpan<char> * length: int -> string member Next: unit -> int + 2 overloads member NextBytes: buffer: byte array -> unit + 1 overload member NextDouble: unit -> float member NextInt64: unit -> int64 + 2 overloads member NextSingle: unit -> float32 member Shuffle<'T> : values: Span<'T> -> unit + 1 overload ...
<summary>Represents a pseudo-random number generator, which is an algorithm that produces a sequence of numbers that meet certain statistical requirements for randomness.</summary>

--------------------
Random() : Random
Random(Seed: int) : Random

val days: int

Multiple items
val int: value: 'T -> int (requires member op_Explicit)

--------------------
type int = int32

--------------------
type int<'Measure> = int

val d: int

DateTime.AddDays(value: float) : DateTime

Random.NextDouble() : float

val min: DateTime

val max: DateTime

val ls: Series<DateTime,float>

type DelayedSeries = static member FromIndexVectorLoader: scheme: IAddressingScheme * vectorBuilder: IVectorBuilder * indexBuilder: IIndexBuilder * min: 'K * max: 'K * loader: Func<'K,BoundaryBehavior,'K,BoundaryBehavior,Task<IIndex<'K> * IVector<'V>>> -> Series<'K,'V> (requires equality) + 1 overload static member FromValueLoader: min: 'K * max: 'K * loader: Func<'K,BoundaryBehavior,'K,BoundaryBehavior,Task<KeyValuePair<'K,'V> seq>> -> Series<'K,'V> (requires comparison) + 1 overload
<summary> This type exposes a single static method `DelayedSeries.Create` that can be used for constructing data series (of type <c>Series<K, V></c>) with lazily loaded data. You can use this functionality to create series that represents e.g. an entire price history in a database, but only loads data that are actually needed. For more information see the [lazy data loading tutorial](../lazysource.html). </summary>
<example> Assuming we have a function <c>generate lo hi</c> that generates data in the specified <c>DateTime</c> range, we can create lazy series as follows: <code> let ls = DelayedSeries.Create(min, max, fun (lo, lob) (hi, hib) -> async { printfn "Query: %A - %A" (lo, lob) (hi, hib) return generate lo hi }) </code> The arguments <c>min</c> and <c>max</c> specify the complete range of the series. The function passed to <c>Create</c> is called with minimal and maximal required key (<c>lo</c> and <c>hi</c>) and with two values that specify boundary behaviour. </example>
<category>Specialized frame and series types</category>

static member DelayedSeries.FromValueLoader: min: 'K * max: 'K * loader: ('K * BoundaryBehavior -> 'K * BoundaryBehavior -> Async<KeyValuePair<'K,'V> seq>) -> Series<'K,'V> (requires comparison)
static member DelayedSeries.FromValueLoader: min: 'K * max: 'K * loader: Func<'K,BoundaryBehavior,'K,BoundaryBehavior,Threading.Tasks.Task<KeyValuePair<'K,'V> seq>> -> Series<'K,'V> (requires comparison)

val lo: DateTime

val lob: BoundaryBehavior

val hi: DateTime

val hib: BoundaryBehavior

val async: AsyncBuilder

val printfn: format: Printf.TextWriterFormat<'T> -> 'T

type BoundaryBehavior = | Inclusive | Exclusive
<summary> Specifies the boundary behavior for the `IIndexBuilder.GetRange` operation (whether the boundary elements should be included or not) </summary>

union case BoundaryBehavior.Inclusive: BoundaryBehavior

val slice: Series<DateTime,float>

val df: Frame<DateTime,string>

val frame: columns: ('a * #ISeries<'c>) seq -> Frame<'c,'a> (requires equality and equality)
<summary> A function for constructing data frame from a sequence of name - column pairs. This provides a nicer syntactic sugar for `Frame.ofColumns`. </summary>
<example> To create a simple frame with two columns, you can write: <code> frame [ "A" => series [ 1 => 30.0; 2 => 35.0 ] "B" => series [ 1 => 30.0; 3 => 40.0 ] ] </code></example>
<category>Frame construction</category>

val slicedDf: Frame<DateTime,string>

property Frame.Rows: RowSeries<DateTime,string> with get
<category>Accessors and slicing</category>