Abstract Background This study aimed to synthesize melatonin laoded chitosan nanoparticles (MEL NPs) and assess its effects on the production of bioactive metabolites, antioxidant activity and gene expression in O. basilicum culture. Results Our results showed that the total phenolic (TPC) and total flavonoid (TFC) content of basil leaves treated with melatonin (MEL) and MEL NPs at 5, 10, and 20 μm was significantly increased, with the rosmarinic acid content also being enhanced. Treatment with 10 µM MEL NPs induced the most pronounced enhancement in total phenolic content (TPC) and total flavonoid content (TFC), reaching 3.48 and 3.79- fold, respectively, compared with the control. meanwhile, the same treatment resulted in the maximum accumulation of rosmarinic acid (RA), 19.73 mg g⁻¹ FW. Furthermore, the application of MEL NPs or melatonin markedly stimulated the phenylprpanoid pathway in basil leaves, as evidenced by the enhanced activities of tyrosine aminotransferase (TAT) and phenylalanine ammonia-lyase (PAL). This enzymatic stimulation was accompanied by significant transcriptional activation of key pathway genes. Notably, basil leaves treated with 10 µM MEL NPs showed pronounced upregulation of 4-coumarate: CoA ligase (4 CL ) and Hydroxyphenylpyruvate reductase ( HPPR ) transcript with expression levels increasing by 3.45- and 5.47-fold, respectively, compared with the untreated control. Conclusion The study concluded that nano encapsulated melatonin at 10 µM was the most effective elicitor for stimulating rosmarinic acid biosynthesis by coordinating transcriptional activation of key biosynthetic genes and enhancing tha activities of related enzymes in basil tissue culture. This is due to its stability and targeted delivery compared to conventional melatonin in large-scale Ocimum basilicum L. plant cultivation as a resource for food and pharmaceutical industries.