any plans? the update skip lock is easily accomplish in mongodb with a transaction
Mongo already has a way of using a collection with a queue, specifically with the findAndModify function. here is some more details here: https://mongodb.github.io/node-mongodb-native/schema/chapter7/
I don’t think you would need to do much work to extend solid queue, i will look into this a bit as i could benefit as well.
Just use a real database, like postgres.
@fatkodima Thanks for your valuable input…
While using real data bases like postgres may be awesome, in this case mongo can give you some bennefits.
1 findAndModify, especially in the current version of mongo is more performant than the postgres FOR UPDATE SKIP LOCKED,You wont need to worry about race condistions under high load.
with postgres, each worker locks rows, which can lead to contention and reduced performance as the number of workers increases.
2 If you get large enough and need to do things like sharding the DB, on mongo, its a one line command, on postgres, its a decent sized effort, especially if you have to do it on a large production DB without downtime. mongo also would support things like regional tagging so your workers in lets say EU can proccess EU data while the US processes US data, you could set this to fail over to each other in the event of an outage so your wont go down if you lose a DC.
3 cost bennefits, while there are tons of companies who will rent you a nice 1cpu 2gb postgres managed db for $50/month… mongo has a managed serverless plan: $0.10 / million reads $1.00 / million writes $0.25 / GB-month transfer.
Lets say you small to medium app has 100k jobs per day, each job has 3 writes and 3 reads, thats a woping $5/month (depending on transfer) and its managed…
For 1) can you provide real-world benchmarks? Where the performance of postgres will be so much worse, that it will matter?
For 2) can you share what scale you operate? In terms of requests, data size, nodes numbers etc.
For 3) - 50$ per month is some fantastic number. Just 5$ on Hetzner (https://www.hetzner.com/cloud/)
Lets say you small to medium app
If you are a small to medium company, then keep it simple and use a real database and not a fake one.
Some solutions are better with a document database instead of relational one. As for a job adapter, using a document database under the hoods for solid queue, it just as useful as a relational database. @fatkodima please remember the discussion guidelines.