Parameter Initialization Task

A Parameter Initialization Task is a Computer Programming Initialization Task that sets initial value for parameters.

• Context:
  • Task Input: parameter initializers.
  • Task Output: parameter estimators.
• Example(s):
  • Parameter Random Initialization Task,
  • Bias Initialization Task,
  • Lipschitz Constrained Parameter Initialization Task,
  • Neural Network Weights Initialization Task.
  • …
• Counter-Example(s):
  • Computer Programming Declaration Task,
  • Computer Programming Deinitialization Task,
  • Lazy Initialization Task.
• See: Machine Code, Computer Programming, Data Object, Programming Language, Run Time, Program Lifecycle, Object-Oriented Programming, Object-Oriented Programming Constructor.

---

---

References

2021a

• (Swift Doc., 2021) ⇒ https://docs.swift.org/swift-book/LanguageGuide/Initialization.html Retrieved:2021-5-23.
  • QUOTE: Initialization is the process of preparing an instance of a class, structure, or enumeration for use. This process involves setting an initial value for each stored property on that instance and performing any other setup or initialization that’s required before the new instance is ready for use.

    You implement this initialization process by defining initializers, which are like special methods that can be called to create a new instance of a particular type. Unlike Objective-C initializers, Swift initializers don’t return a value. Their primary role is to ensure that new instances of a type are correctly initialized before they’re used for the first time.

    Instances of class types can also implement a deinitializer, which performs any custom cleanup just before an instance of that class is deallocated. For more information about deinitializers, see Deinitialization.

2021

• (Srihari, 2021) ⇒ Sargur N. Srihari (2021). “Parameter Initialization Strategies" Retrieved:2021-5-23.
  • QUOTE: Keras Initialization.
    • Initializations define the way to set the initial random weights of Keras layers.
    • The keyword arguments used for passing initializers to layers will depend on the layer.
    • Usually it is simply kernel_initializer and bias_initializer:

      model.add (Dense(64, kernel_initializer="randon_uniforn’, bias_initializer='zeros'))

2020

• (Xu et al., 2019) ⇒ Hongfei Xu, Qiuhui Liu, Josef van Genabith, Deyi Xiong, and Jingyi Zhang (2020). "Lipschitz Constrained Parameter Initialization for Deep Transformers". In: Proceedings of the 58th Annual Meeting of the Association for Computational Linguistics (ACL 2020).
  • QUOTE: We then compare the subtle differences in computation order in considerable detail, and present a parameter initialization method that leverages the Lipschitz constraint on the initialization of Transformer parameters that effectively ensures training convergence. In contrast to findings in previous research we further demonstrate that with Lipschitz parameter initialization, deep Transformers with the original computation order can converge, and obtain significant BLEU improvements with up to 24 layers.

---