The Theory behind Vortex Generators
I reckon the effect of vortex generators on aircraft wings was one of those innovations that was 'discovered' first and explained later.
Vortex Generators (VGs) are small plastic or metal fins that can be placed aft of the leading edge of the wing and tail on STOL aircraft. They work by controlling the airflow over the surface of the wing by creating vortices that energise the boundary layer. The best vortex forms when airflow progressively spills over an angled leading edge creating a slim, tight, tidy vortex that stays close to the wing surface, all the way to the trailing edge.
The result is a wing that can fly at significantly higher angles of attack before the onset of boundary layer separation and can therefore achieve a significantly higher maximum lift coefficient. When mounted on the wings, VGs reduce stall speed and increase climb capability. When mounted on the vertical tail, they increase rudder effectiveness and lower Vmc.
The 'Rip‑Stop' Effect
This is an effect that I've noticed by experience with video tuft testing VGs in action. I’ve never seen it written up before, except for an aside mentioned once in the results of some wind tunnel testing. It noted “the VGs seem to help prevent a disturbance from spreading” In effect, the VGs act somewhat like miniature stall fences.
I call it the 'Rip‑Stop' effect as an analogy to nylon fabric. When thin nylon starts to tear, the rip can zip across very easily, so manufacturers add stronger threads about every 6mm to stop it spreading. That appears to be exactly what the VGs do to a spreading stall disruption.
The vortices seem to 'tie' the airflow together like those rip‑stop threads, keeping it in a coherent sheet flow, well attached to the wing surface, rather than allowing a stall disturbance to spread. The higher angle of attack is due to retaining the boundary layer, but the control of stall spread is due to the 'Rip‑Stop' effect.