paxblind.blogg.se - Python spawn subprocess in background

Depending on the size of data you might need to use a RDMS database or data platform to offload processing to.

You are going to get a a lot of time penalty costs reading and writing to a SQLite file (file I/O is the slowest of all).

So an example might be, year/state/zipcode/company or company/year/local/department/project etc… The partitioning really depends on what you need to accomplish. Columns with low unique values are a good place to look for that. Then sub-partition it based on dimensions.

If you have a file with 10 years of time series but you typically only use 3 years. Partition your data so you’re only processing what you need to.For instance are you running an aggregation on a set of data that can be filtered first? Review your query / data manipulation for items that are slowing it down.This list is the order, I’d investigate in. The tricky part identifiying what is the best solution for your problem. Here is where I’d look to solve the problem.ĭon’t let this list overwhelm you… Most of these are less then a days work to setup even if you don’t have much exposure to them.

At best what you can implement with Dash is a async workflow process that says, we’re processing the data we’ll send you an alert/notification when it’s done. What you’re describing is a data processing problem, which is not going to be handled well by any web framework. Dash is a web framework that is managing the presentation and inter-actions. IMO this doesn’t sound like a Dash issue and you’ll make a mess if you try to use Dash to solve it. What modules you’re using (other than Dash) anything that gives the full picture.

Can you add in more such how much data you’re processing, what type of system it’s running on (cores, mem, cloud provider), what type of calculations you are running. There is a a lot of missing information so it’s not possible to tell you what the best approach will be for your problem.