If defrag is taking more than a few minutes, then you're not doing it often enough, or using the right tool. Good, dedicated defrag tools can be scheduled to run as needed and run in low priority.

Windows, by default, sets up the OS to fragment your drive over time. Especially if you don't have enough RAM, the fragmentation is guaranteed to happen and eventually cripple your drive. Fragmented MFT and paging files are the rule with Windows, not an exception.

Oddly enough, a fresh install of Windows and applications can result in a badly fragmented drive right off the bat, and the problems just multiply from there.

As to the OP's problem with a slow system, anything is possible and some basic hardware troubleshooting is needed. If rebooting fixes it, then memory leaks or inadequate RAM amount are possible, but a fragmented drive could be involved as well. It's rarely just one thing.