Webpacker recompiles everything

I discussed the problem I am facing in this issue, the main issue I believe is that webpacker recompiles everything even if only one file is added, which leads to a lot of unnecessary compile time. Is there is a way to make it only new or only modified files ?