Support for cloth self-collision is limited and using multi-layered cloth can be unstable.
The avatar for the cloth-simulation should not have more than 15k – 20k vertices. 5k – 7k vertices is recommended for optimal performance. If your avatar has a higher number of vertices, please use the hidden avatar functionality (Collider mesh) (see e.g. https://youtu.be/lQ07GfuKP-4?t=143) It can simply be a low-poly mesh (e.g. simplified face, arms and feet) or the same mesh but with a lower Level of Details (LOD).