PySpark

This tutorial will cover a complete worked example of how to stream data in PySpark using the applyInPandasWithState function and foreachBatch. Spark structured streaming does not always come with the tools needed out-of-the-box and by using applyInPandasWithState and foreachBatch the streaming functionality can be  customised. The example will use the scenario of streaming data from […]