Of Notes
Towards a Theory of Dynamical Transferability
Table of Contents
Can we predict which unfamiliar dynamical systems will provide useful pretraining knowledge for a data-sparse target, before spending resources training on them?
- Diversification: Explore alternative pretraining domains as an action space, identifying systems with potentially transferable dynamical structures.
- Reintegration: Transfer knowledge from these domains into a foundation model specialised for a data-sparse target system.
- Adaptive generalisation: Iteratively search, diversify and reintegrate, using improvements in target performance to guide subsequent exploration.
Specification
A compute-constrained, target-feedback-driven curriculum for repeated diversification and reintegration, designed to discover transferable structure across heterogeneous dynamical domains.
Can iterative, curriculum-guided diversification and reintegration improve data-efficient generalisation in foundation models of dynamical systems compared with conventional sequential pre-training and fine-tuning under equivalent computational budgets?
Questions
- Can we identify shared dynamical characteristics across different systems that predict successful cross-domain transfer, and how do model architectures and training strategies influence this?
- Can we learn transferable representations of underlying dynamics from sparse, censored, or otherwise incomplete observations?
- (Diversification~Reintegration, strong results on Cellular Automata)
- Can these principles improve the pretraining and generalisation of foundation models, particularly in data-constrained domains such as neuroscience (e.g. fMRI)?