Delay-loaded series

The DelayedSeries type provides an efficient way to create series whose data is loaded on-demand. For example, you may have a large time series stored in a CSV file or in a database and you do not want to load all the data in memory if the user only needs a small part of it.

When you create a delayed series, you specify the overall range of the series (i.e. the minimum and maximum key value) and you provide a function that loads a specified sub-range of the series. When the user accesses a continuous range of the series, the loading function is called to retrieve the data.

Creating a delayed series

To create a delayed series, we need a function that generates data for a given range. The following function generates a series with random data for a given date range with a day frequency:

let generate (low:DateTime) (high:DateTime) : seq<KeyValuePair<DateTime,float>> = 
    let rnd = Random()
    let days = int (high - low).TotalDays
    seq [ for d in 0 .. days -> KeyValuePair(low.AddDays(float d), rnd.NextDouble()) ]

Now we use DelayedSeries.FromValueLoader to create a delayed series. It takes the overall minimum and maximum key of the series and a function that loads data for a sub-range. The loading function gets the lower and upper bound as a tuple of (key, BoundaryBehavior) values where BoundaryBehavior is either Inclusive or Exclusive:

let min = DateTime(2010, 1, 1)
let max = DateTime(2013, 1, 1)

let ls = DelayedSeries.FromValueLoader(min, max, fun (lo, lob) (hi, hib) -> async {
    printfn "Query: %A - %A" lo hi
    let lo = if lob = BoundaryBehavior.Inclusive then lo else lo.AddDays(1.0)
    let hi = if hib = BoundaryBehavior.Inclusive then hi else hi.AddDays(-1.0)
    return generate lo hi })

The key thing about the above is that, so far, no data has been loaded. The loading function is called only when we access part of the series.

Slicing and using delayed series

We can now use the series as usual - for example, to get data for the entire year 2012:

let slice = ls.[DateTime(2012, 1, 1) .. DateTime(2012, 12, 31)]
slice

val slice: Series<DateTime,float> =
  
(Delayed series [01/01/2012 .. 12/31/2012]) 

val it: Series<DateTime,float> =
  
(Delayed series [01/01/2012 .. 12/31/2012])

Similarly, we can add the delayed series to a data frame. When doing this, Deedle will only load the data that is needed. In the following example, we add the series to a frame and then access only a slice:

let df = frame ["Values" => ls]
let slicedDf = df.Rows.[DateTime(2012,6,1) .. DateTime(2012,6,30)]
slicedDf

Query: 01/01/2010 00:00:00 - 01/01/2013 00:00:00
Query: 06/01/2012 00:00:00 - 06/30/2012 00:00:00
val df: Frame<DateTime,string> =
  
              Values              
01/01/2010 -> 0.10521580404025521 
01/02/2010 -> 0.46897471866253293 
01/03/2010 -> 0.9676123159239752  
01/04/2010 -> 0.7098231224409021  
01/05/2010 -> 0.5112773349635729  
01/06/2010 -> 0.771789340985794   
01/07/2010 -> 0.6786292729396111  
01/08/2010 -> 0.6208362910117433  
01/09/2010 -> 0.10370320612212891 
01/10/2010 -> 0.6071882252839856  
01/11/2010 -> 0.2557779296358704  
01/12/2010 -> 0.7608545848800411  
01/13/2010 -> 0.5191624650226938  
01/14/2010 -> 0.4927662920571717  
01/15/2010 -> 0.7702486105998264  
:             ...                 
12/18/2012 -> 0.6157860347056444  
12/19/2012 -> 0.24374920052989213 
12/20/2012 -> 0.25157255701468584 
12/21/2012 -> 0.48910010836202833 
12/22/2012 -> 0.9809746832731672  
12/23/2012 -> 0.41022427914065973 
12/24/2012 -> 0.21817802341071046 
12/25/2012 -> 0.7477739338344829  
12/26/2012 -> 0.145190418748134   
12/27/2012 -> 0.02966522095139701 
12/28/2012 -> 0.635852974147087   
12/29/2012 -> 0.6901146554663987  
12/30/2012 -> 0.29671458120110705 
12/31/2012 -> 0.3678921575853009  
01/01/2013 -> 0.9788560881149595  

val slicedDf: Frame<DateTime,string> =
  
              Values              
06/01/2012 -> 0.18983785904408945 
06/02/2012 -> 0.8986569866130855  
06/03/2012 -> 0.8030877525531345  
06/04/2012 -> 0.39883736722171126 
06/05/2012 -> 0.18382509866244745 
06/06/2012 -> 0.18208633530351437 
06/07/2012 -> 0.23185694414610958 
06/08/2012 -> 0.7921824503190124  
06/09/2012 -> 0.7337080680490042  
06/10/2012 -> 0.7904660256421128  
06/11/2012 -> 0.06614038108704856 
06/12/2012 -> 0.40825521401750897 
06/13/2012 -> 0.15915824471579865 
06/14/2012 -> 0.7192091536321475  
06/15/2012 -> 0.5020346474194276  
06/16/2012 -> 0.5000807862700559  
06/17/2012 -> 0.4538024278322519  
06/18/2012 -> 0.5727384531827907  
06/19/2012 -> 0.7755097132860543  
06/20/2012 -> 0.02667329787298134 
06/21/2012 -> 0.08416298445168924 
06/22/2012 -> 0.02151203610683483 
06/23/2012 -> 0.8105393103757411  
06/24/2012 -> 0.553833155999256   
06/25/2012 -> 0.12285178070362679 
06/26/2012 -> 0.1358008466084205  
06/27/2012 -> 0.651141305529562   
06/28/2012 -> 0.2286778717795186  
06/29/2012 -> 0.20489651311905865 
06/30/2012 -> 0.7299793750250709  

val it: Frame<DateTime,string> =
  
              Values              
06/01/2012 -> 0.18983785904408945 
06/02/2012 -> 0.8986569866130855  
06/03/2012 -> 0.8030877525531345  
06/04/2012 -> 0.39883736722171126 
06/05/2012 -> 0.18382509866244745 
06/06/2012 -> 0.18208633530351437 
06/07/2012 -> 0.23185694414610958 
06/08/2012 -> 0.7921824503190124  
06/09/2012 -> 0.7337080680490042  
06/10/2012 -> 0.7904660256421128  
06/11/2012 -> 0.06614038108704856 
06/12/2012 -> 0.40825521401750897 
06/13/2012 -> 0.15915824471579865 
06/14/2012 -> 0.7192091536321475  
06/15/2012 -> 0.5020346474194276  
06/16/2012 -> 0.5000807862700559  
06/17/2012 -> 0.4538024278322519  
06/18/2012 -> 0.5727384531827907  
06/19/2012 -> 0.7755097132860543  
06/20/2012 -> 0.02667329787298134 
06/21/2012 -> 0.08416298445168924 
06/22/2012 -> 0.02151203610683483 
06/23/2012 -> 0.8105393103757411  
06/24/2012 -> 0.553833155999256   
06/25/2012 -> 0.12285178070362679 
06/26/2012 -> 0.1358008466084205  
06/27/2012 -> 0.651141305529562   
06/28/2012 -> 0.2286778717795186  
06/29/2012 -> 0.20489651311905865 
06/30/2012 -> 0.7299793750250709

namespace System

namespace System.Collections

namespace System.Collections.Generic

namespace Deedle

namespace Deedle.Indices

val fsi: FSharp.Compiler.Interactive.InteractiveSession

member FSharp.Compiler.Interactive.InteractiveSession.AddPrinter: ('T -> string) -> unit

val o: obj

type obj = Object

val iface: Type

Object.GetType() : Type

val fmt: Reflection.MethodInfo

Type.GetMethod(name: string) : Reflection.MethodInfo
   (+0 other overloads)
Type.GetMethod(name: string, types: Type array) : Reflection.MethodInfo
   (+0 other overloads)
Type.GetMethod(name: string, bindingAttr: Reflection.BindingFlags) : Reflection.MethodInfo
   (+0 other overloads)
Type.GetMethod(name: string, types: Type array, modifiers: Reflection.ParameterModifier array) : Reflection.MethodInfo
   (+0 other overloads)
Type.GetMethod(name: string, bindingAttr: Reflection.BindingFlags, types: Type array) : Reflection.MethodInfo
   (+0 other overloads)
Type.GetMethod(name: string, genericParameterCount: int, types: Type array) : Reflection.MethodInfo
   (+0 other overloads)
Type.GetMethod(name: string, genericParameterCount: int, types: Type array, modifiers: Reflection.ParameterModifier array) : Reflection.MethodInfo
   (+0 other overloads)
Type.GetMethod(name: string, genericParameterCount: int, bindingAttr: Reflection.BindingFlags, types: Type array) : Reflection.MethodInfo
   (+0 other overloads)
Type.GetMethod(name: string, bindingAttr: Reflection.BindingFlags, binder: Reflection.Binder, types: Type array, modifiers: Reflection.ParameterModifier array) : Reflection.MethodInfo
   (+0 other overloads)
Type.GetMethod(name: string, bindingAttr: Reflection.BindingFlags, binder: Reflection.Binder, callConvention: Reflection.CallingConventions, types: Type array, modifiers: Reflection.ParameterModifier array) : Reflection.MethodInfo
   (+0 other overloads)

Reflection.MethodBase.Invoke(obj: obj, parameters: obj array) : obj
Reflection.MethodBase.Invoke(obj: obj, invokeAttr: Reflection.BindingFlags, binder: Reflection.Binder, parameters: obj array, culture: Globalization.CultureInfo) : obj

Multiple items
val string: value: 'T -> string

--------------------
type string = String

val generate: low: DateTime -> high: DateTime -> KeyValuePair<DateTime,float> seq

val low: DateTime

Multiple items
type DateTime = new: date: DateOnly * time: TimeOnly -> unit + 16 overloads member Add: value: TimeSpan -> DateTime member AddDays: value: float -> DateTime member AddHours: value: float -> DateTime member AddMicroseconds: value: float -> DateTime member AddMilliseconds: value: float -> DateTime member AddMinutes: value: float -> DateTime member AddMonths: months: int -> DateTime member AddSeconds: value: float -> DateTime member AddTicks: value: int64 -> DateTime ...
<summary>Represents an instant in time, typically expressed as a date and time of day.</summary>

--------------------
DateTime ()
   (+0 other overloads)
DateTime(ticks: int64) : DateTime
   (+0 other overloads)
DateTime(date: DateOnly, time: TimeOnly) : DateTime
   (+0 other overloads)
DateTime(ticks: int64, kind: DateTimeKind) : DateTime
   (+0 other overloads)
DateTime(date: DateOnly, time: TimeOnly, kind: DateTimeKind) : DateTime
   (+0 other overloads)
DateTime(year: int, month: int, day: int) : DateTime
   (+0 other overloads)
DateTime(year: int, month: int, day: int, calendar: Globalization.Calendar) : DateTime
   (+0 other overloads)
DateTime(year: int, month: int, day: int, hour: int, minute: int, second: int) : DateTime
   (+0 other overloads)
DateTime(year: int, month: int, day: int, hour: int, minute: int, second: int, kind: DateTimeKind) : DateTime
   (+0 other overloads)
DateTime(year: int, month: int, day: int, hour: int, minute: int, second: int, calendar: Globalization.Calendar) : DateTime
   (+0 other overloads)

val high: DateTime

Multiple items
val seq: sequence: 'T seq -> 'T seq

--------------------
type 'T seq = IEnumerable<'T>

Multiple items
type KeyValuePair = static member Create<'TKey,'TValue> : key: 'TKey * value: 'TValue -> KeyValuePair<'TKey,'TValue>
<summary>Creates instances of the <see cref="T:System.Collections.Generic.KeyValuePair`2" /> struct.</summary>

--------------------
type KeyValuePair<'TKey,'TValue> = new: key: 'TKey * value: 'TValue -> unit member Deconstruct: key: byref<'TKey> * value: byref<'TValue> -> unit member ToString: unit -> string member Key: 'TKey member Value: 'TValue
<summary>Defines a key/value pair that can be set or retrieved.</summary>
<typeparam name="TKey">The type of the key.</typeparam>
<typeparam name="TValue">The type of the value.</typeparam>

--------------------
KeyValuePair ()
KeyValuePair(key: 'TKey, value: 'TValue) : KeyValuePair<'TKey,'TValue>

Multiple items
val float: value: 'T -> float (requires member op_Explicit)

--------------------
type float = Double

--------------------
type float<'Measure> = float

val rnd: Random

Multiple items
type Random = new: unit -> unit + 1 overload member GetHexString: stringLength: int * ?lowercase: bool -> string + 1 overload member GetItems<'T> : choices: ReadOnlySpan<'T> * length: int -> 'T array + 2 overloads member GetString: choices: ReadOnlySpan<char> * length: int -> string member Next: unit -> int + 2 overloads member NextBytes: buffer: byte array -> unit + 1 overload member NextDouble: unit -> float member NextInt64: unit -> int64 + 2 overloads member NextSingle: unit -> float32 member Shuffle<'T> : values: Span<'T> -> unit + 1 overload ...
<summary>Represents a pseudo-random number generator, which is an algorithm that produces a sequence of numbers that meet certain statistical requirements for randomness.</summary>

--------------------
Random() : Random
Random(Seed: int) : Random

val days: int

Multiple items
val int: value: 'T -> int (requires member op_Explicit)

--------------------
type int = int32

--------------------
type int<'Measure> = int

val d: int

DateTime.AddDays(value: float) : DateTime

Random.NextDouble() : float

val min: DateTime

val max: DateTime

val ls: Series<DateTime,float>

type DelayedSeries = static member FromIndexVectorLoader: scheme: IAddressingScheme * vectorBuilder: IVectorBuilder * indexBuilder: IIndexBuilder * min: 'K * max: 'K * loader: Func<'K,BoundaryBehavior,'K,BoundaryBehavior,Task<IIndex<'K> * IVector<'V>>> -> Series<'K,'V> (requires equality) + 1 overload static member FromValueLoader: min: 'K * max: 'K * loader: Func<'K,BoundaryBehavior,'K,BoundaryBehavior,Task<KeyValuePair<'K,'V> seq>> -> Series<'K,'V> (requires comparison) + 1 overload
<summary> This type exposes a single static method `DelayedSeries.Create` that can be used for constructing data series (of type <c>Series<K, V></c>) with lazily loaded data. You can use this functionality to create series that represents e.g. an entire price history in a database, but only loads data that are actually needed. For more information see the [lazy data loading tutorial](../lazysource.html). </summary>
<example> Assuming we have a function <c>generate lo hi</c> that generates data in the specified <c>DateTime</c> range, we can create lazy series as follows: <code> let ls = DelayedSeries.Create(min, max, fun (lo, lob) (hi, hib) -> async { printfn "Query: %A - %A" (lo, lob) (hi, hib) return generate lo hi }) </code> The arguments <c>min</c> and <c>max</c> specify the complete range of the series. The function passed to <c>Create</c> is called with minimal and maximal required key (<c>lo</c> and <c>hi</c>) and with two values that specify boundary behaviour. </example>
<category>Specialized frame and series types</category>

static member DelayedSeries.FromValueLoader: min: 'K * max: 'K * loader: ('K * BoundaryBehavior -> 'K * BoundaryBehavior -> Async<KeyValuePair<'K,'V> seq>) -> Series<'K,'V> (requires comparison)
static member DelayedSeries.FromValueLoader: min: 'K * max: 'K * loader: Func<'K,BoundaryBehavior,'K,BoundaryBehavior,Threading.Tasks.Task<KeyValuePair<'K,'V> seq>> -> Series<'K,'V> (requires comparison)

val lo: DateTime

val lob: BoundaryBehavior

val hi: DateTime

val hib: BoundaryBehavior

val async: AsyncBuilder

val printfn: format: Printf.TextWriterFormat<'T> -> 'T

type BoundaryBehavior = | Inclusive | Exclusive
<summary> Specifies the boundary behavior for the `IIndexBuilder.GetRange` operation (whether the boundary elements should be included or not) </summary>

union case BoundaryBehavior.Inclusive: BoundaryBehavior

val slice: Series<DateTime,float>

val df: Frame<DateTime,string>

val frame: columns: ('a * #ISeries<'c>) seq -> Frame<'c,'a> (requires equality and equality)
<summary> A function for constructing data frame from a sequence of name - column pairs. This provides a nicer syntactic sugar for `Frame.ofColumns`. </summary>
<example> To create a simple frame with two columns, you can write: <code> frame [ "A" => series [ 1 => 30.0; 2 => 35.0 ] "B" => series [ 1 => 30.0; 3 => 40.0 ] ] </code></example>
<category>Frame construction</category>

val slicedDf: Frame<DateTime,string>

property Frame.Rows: RowSeries<DateTime,string> with get
<category>Accessors and slicing</category>