You can use Scripting tiles to automate data processing and data science workflows. For an overview, see New Magic ETL Tiles: Scripting.
In this article, we’ll provide an overview of troubleshooting tips for optimizing memory use in your container. In some cases, you may still need to upgrade your container. For guidance on requesting a container upgrade, see Requesting Memory Upgrades.
You can use Magic ETL tiles to reduce the memory footprint in your script. Magic ETL tiles have been optimized for memory-intensive operations. We recommend excluding intensive operations in your script that can be handled through Magic ETL tiles instead.
You can use the Performance tiles in Magic ETL to exclude columns in your DataSet that are not needed in your script. For information on using Performance tiles, see New Magic ETL Tiles: Performance. This will reduce the DataSet that will be processed in your script.
You can also use the Filter Rows tile and the Remove Duplicates tile to exclude rows that are not needed for the DataSet used in your script. For information on using the Filter tile, see New Magic ETL Tiles: Filter.
For a comprehensive look at operations that you can handle with Magic Tiles, see New Magic ETL.
Verify that you are using the latest versions of the packages needed in your script. Newer versions of the package are often optimized for performance and memory. If you need to request a package upgrade, see Requesting Custom Environments for details.
Breaking up an operation into segments can help reduce memory footprint, especially for intensive operations like looping. You could also consider breaking up the DataFlow into separate DataFlows.
Test memory for data size only. You can set up a DataFlow that simply writes the input (no operations in the script.) If the DataFlow, errors out then you need to reduce the DataSet size (Tip 1) or evaluate a memory upgrade.
Requesting Memory Upgrades
While the tips above may help lower your memory footprint, in some cases your workflow will simply require more memory. To request this upgrade, see Requesting Memory Upgrades for a walkthrough.