Neural Architecture Search

Neural Architecture Search has emerged as a transformative paradigm for automating the design of deep neural networks, reducing the reliance on human expertise and enabling the discovery of architectures that often surpass manually designed counterparts. This book presents a comprehensive survey of NAS methodologies, tracing the evolution from early reinforcement learning-based and evolutionary approaches to modern differentiable and one-shot methods. The book systematically categorizes NAS techniques along three fundamental dimensions: search space design, search strategy, and performance estimation. A unified mathematical framework is introduced that encompasses gradient-based, reinforcement learning-based, and evolutionary NAS under a common optimization objective. Furthermore, hardware-aware NAS is examined, which integrates latency, energy consumption, and memory constraints into the architecture search process. The theoretical foundations of differentiable NAS are analyzed, including the bilevel optimization problem and its convergence properties, and recent advances in addressing instability and fairness issues are discussed. The book also surveys applications across computer vision, natural language processing, and scientific computing, and concludes with a critical analysis of open challenges, including computational efficiency, reproducibility, and the need for domain-specific search spaces, proposing directions for future research.