You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
How do you handle cases when data is too much for memory? Pandas issue #254
Hello @nyck33
visualpython is using python libraries such as pandas, numpy and so on. So it'll work as other libraries do.
And some more explanation is needed for your second line.
So how are you loading it in say am making queries to Snowflake?
@minjk-bl Pandas can only hold as much data as the amount of memory on machines. What if there is a dataset with millions of rows that takes 10 GB and I only have 8 GB RAM on my machine? This is where Spark and Dask dataframes are useful right? But if you're trying to load with Pandas then it's just going to be an error. How do you handle this? Do you load in chunks and say for example: "rows 0 to 10,000", "10,001 to 20,000", etc?
@nyck33
My apologies for the delayed response.
In fact, we are not handling large-scale data processing separately.
Visual Python is currently running on the configured Jupyter environment using Python packages, so no specific code optimization has been implemented.
Activity
minjk-bl commentedon Nov 27, 2023
Hello @nyck33
visualpython is using python libraries such as pandas, numpy and so on. So it'll work as other libraries do.
And some more explanation is needed for your second line.
nyck33 commentedon Nov 27, 2023
@minjk-bl Pandas can only hold as much data as the amount of memory on machines. What if there is a dataset with millions of rows that takes 10 GB and I only have 8 GB RAM on my machine? This is where Spark and Dask dataframes are useful right? But if you're trying to load with Pandas then it's just going to be an error. How do you handle this? Do you load in chunks and say for example: "rows 0 to 10,000", "10,001 to 20,000", etc?
minjk-bl commentedon Feb 26, 2024
@nyck33
My apologies for the delayed response.
In fact, we are not handling large-scale data processing separately.
Visual Python is currently running on the configured Jupyter environment using Python packages, so no specific code optimization has been implemented.