A "meshlet" is a concept built around the specific architecture of how shaders work.
Shaders are invoked in work groups, even when shader stages don't explicitly represent that concept in the API. This means that some number of invocations are executed in lock-step with each other on the same core. That's the natural way they work.
In theory this could allow different invocations in the same work group to share data. This is what compute shaders and tessellation control shaders do with shared
and out
variables respectively.
However, the nature of the traditional vertex processing pipeline makes this data sharing difficult. The order of execution of vertex shaders, which invocations go into which work groups, can change based on how indices are processed, the state of the post-T&L vertex cache, etc. It is not reliable, and therefore you cannot really share information between VS invocations.
The mesh shader pipeline is designed to more directly mirror hardware. In this pipeline, there are no equivalents to vertex attributes; if you want to get some vertex data, the shader must use SSBOs or similar tools to fetch them. That is, everything is up to you.
The goal here is to process meshes in work groups. For a mesh of a given size, you execute some number of work groups on that mesh, with each group having some number of invocations that can share data between them.
However, since work group sizes and shared storage is finite and quite small, it is reasonable for a work group to only operate on a portion of the mesh rather than the whole thing. To do this, the mesh needs to be broken down into appropriately sized chunks. We call these chunks "meshlets".