When this works cleanly
The cleanest case is when there are no mods installed yet. You created a Forge instance, never added a mod, want to switch to Fabric, change loader and version, done. Forge's installer is left behind on disk but doesn't interfere.
The second-cleanest case is switching between very similar loaders, Forge to NeoForge, or Fabric to Quilt. Many mods are source-compatible across those pairs, and a fair number of Forge mods just work on NeoForge.
When this gets messy
Cross-family switches (Forge → Fabric, Fabric → Forge, etc.) need the mods folder cleaned out, because Forge mods literally can't load under Fabric and vice versa. The instance will boot but every mod is silently ignored.
Modpack instances that were locked are an extra constraint: the lock blocks Addons tab changes, but the Edit modal still lets you change the loader. That's a footgun, the pack manifest expects a specific loader, switching it leaves the manifest pointing at the wrong loader. If you really want to switch on a modpack instance, open the instance's Settings tab and click Unpair in the Modpack Info section first, so the launcher stops trying to track the pack manifest.
What stays, what changes
Stays: worlds, screenshots, options.txt, your accounts, the instance name and icon, the Minecraft version. Changes: the mod loader and its version. The mods folder doesn't get touched, you need to clean it up.
If the launch fails after switching
Most common cause: old loader's mods are still in instance/mods/. Right-click the instance → Open Folder, go into instance/mods/, delete every JAR. Then add the mods you want for the new loader via the Addons tab.
Second most common: the new loader version doesn't actually exist for the Minecraft version. Some loaders skip versions (Fabric rarely does this, Forge sometimes does for older minor versions). Pick the closest available one.
An alternative: clone first, switch the clone
If you want to keep your Forge setup and try Fabric on the same Minecraft version, right-click the instance → Duplicate, then switch the loader on the duplicate. Both stay around; you can decide which to keep later.