You should start by reviewing the OTC drug monographs that FDA has established for each class (therapeutic categories) of products.
OTC drug monographs are a kind of "recipe book" covering acceptable ingredients, doses, formulations, labeling, and testing. OTC drug monographs are continually updated to add additional ingredients and labeling as needed. Products conforming to a monograph may be marketed without FDA pre-approval, while those that do not must undergo separate review and approval through the New Drug Application (NDA) process.
If you plan to use a new ingredient that has not yet been marketed then you must follow the NDA process, not the monograph process. For example, a drug that was first available only by prescription must be approved through the NDA process. The "switch" of the prescription only drug to OTC status is also via the NDA process.