Gel pens
Gel ink is thick and pigmented. It lays down a bold, saturated line that dries to a crisp edge - headings and hand-lettering hold their shape without feathering. The Uni-Ball Signo and Paper Mate Inkjoy are the benchmarks. The downside: gel cartridges run out faster than rollerball or fineliner ink, and budget gel pens can skip on some papers.
Fineliners
Fineliners use water-based ink and a felt or fibre tip. The line width is fixed and consistent - ideal for ruled notes, annotation and any work where you want a predictable 0.3 or 0.5mm line. They are not as bold as a gel at the same nominal width, but they are more precise. Caps need to go back on or the tip dries.
Rollerballs
A rollerball uses liquid ink on a ball bearing, like a ballpoint - but the ink is water-based and flows more freely. The result is somewhere between a gel and a fountain pen: wetter than a gel, more precise than a fineliner for broad strokes. The Uni-Ball Eye is a well-regarded example. Rollerballs work well on most paper and write without pressure.
Which to choose
For headings and layout in a notebook: a gel. For fine detail, annotation and ruled lines: a fineliner. For a smooth everyday writer that works on any paper without thought: a rollerball. Most journallers end up using all three for different purposes in the same notebook.