A fractal is any equation or pattern that when seen as an image produces a picture which can be zoomed into infinity and will still produce the same picture. It can be cut into parts which look quite like a smaller version of the set that was started with. A simple example is a tree that branches infinitely into smaller branches with those smaller branches branching into even smaller branches and so on.
Fractals are related to the field of psychonautics due to the fact that they are an extremely common component embedded within the visual experience of psychedelic geometry. Fractal images with mathematics are generated through recursive formulas that infinitely repeat an image over itself in a self-similar way.
It could therefore be speculated that the fractals one sees within psychedelic geometry are generated through a similar recursive looping system which is reinterpreting the same visual stimuli multiple times on top of itself, thus causing geometry to flower out into distinctive fractal structures.